Introduction, Elastic Compute Cloud (EC2), Simple Storage Service (S3), Elastic MapReduce (EMR), Amazon Web Services (AWS), AWS Educate Program, Console home, creating a bucket, adding objects to bucket, creating ECE Key pair, pem file, sample Amazon EMR cluster, quick cluster configuration options, preparing sample data and script, sample hive script, pending->running->completed, resetting the environment, terminating the cluster, runing the WordCount.jar in Amazon EMR, AWS command line interface (for windows), listing a public S3 folder using AWS cli.
Hadoop Java Code to process a Chess Dataset (from FICS Games Database in PGN format). It was executed on Amazon EMR. Provisioning of the Hadoop cluster, running/terminating jobs and handling data transfer between EC2(VM) and S3(Object Storage) are automated by Elastic MapReduce.
OS level virtualization, Docker, groups, Linux namespaces, CoreOS, LXC, OpenVZ, Kubernetes, Union mounting, aufs, Docker architecture, Docker client, Docker images, Docker containers, Docker index, DotCloud, basic docker commands, exposing a port in Docker, naming a container, sopping a running container, attaching to a running container, Ctrl+P+Q, /var director, long and short ID of containers, local port and container port, contaiers and images, dockerfile.
Cloud Native Computing Foundation (CNCF), GIFEE (Google Infrastructure For Everyone Else), Kubernetes, SRE: how software engineers do operations, systemd init system, CoreOS, update-engine, etcd, chubby, consensus protocol, CoreOS partitioning, service discovery (etcd), 'The Datacenter as a Computer', lmctfy, mesos containers, passive and active boot paritions in CoreOS, kexec, how etcd works, Leader and Follower, job scheduling (fleet, mesos, kubernetes, swarm), locksmit (reboot manager), service discovery, Gentoo Linux, update distribution in CoreOS, use of semaphores in locksmith, Omaha protocol - open-source version of Google update, ebuild, CoreUpdate, CoreOS cluster architecture, Raft consensus algorithm, ACI (App Container Image), Rocker (rkt), cAdvisor, Kubernetes explained, google container engine, ELK (Elasticsearch, Logstash, Kibana), Project Atomic, Atomic Host, App and Nulecule, Atomic Registry, Rancher - Container Managment Platform, RancherOS.
Kubernetes, containers, pod, replication controllers, service, volumes, namespace, the new stack, decoupling of the application from the lower level, kubernetes architecture, kubelet daemon, labels: key-value pairs, pod example, shared memory, labels, reconciliation loop, current state and the desired state, create, health check, delete, microservices, label query, docker containers, example json configuration template file, google container engine, Docker, starting continers from an existing image, connecting to a container, creating your own docker file, building an image, sharing your images, docker engine, docker's architecture, docker daemon, dockerfile, read-write layer added to the top, namespaces and control groups, union file system, container format, installing docker on Ubuntu, creating docker group, enabling ufw forwarding, configuring dns server for use by Docker, configuring docker to start on boot, verifying your installation with 'docker run hello-world', 'docker ps -a', whalesay example, writing your dockerfile -- updating the whalesay example, building image from docker file, learn about the build process, create a docker-hub account and repository, tag, push and pull your image, containerized app image, docker layers, immutability of docker containers, minimized drift, true multicloud portability.
Introduction to Object storage, metadata inclusion, OSD (Object Based Storage), EMC Atmos, Archive storage, Hybrid storage, Scality, OSD v1 and v2, difference between key-value and object stores, cloud storage, advantages and disadvantages, eventual consistency, strong eventual consistency, CAP theorem, Openstack object storage (swift), object key and metadata, object key naming guidelines in S3, you can't create buckets inside buckets, object metadata in S3, searching on Amazon S3 bucket, content addressable storage, content-addressed vs. location-addressed, Centera system, block and file storage, car parking analogy of object storage.
Virtualization and Cloud
Last Updated: Jun 2018