I use Docker and Docker Compose a lot. Like, every day. It’s a fantastic way to build repeatable demos and examples, that can be torn down and spun up in a repeatable way. But…what happens when the demo that was working is spun up and then tail spins down in a blaze of flames?
Here’s the excerpt of a log from my Oracle container:
$ docker-compose up -d ; docker-compose logs -f oracle
Recreating connect-jdbc_oracle_1_4be0ad4479f8 ...
connect-jdbc_mssql_1_eb2a015c90b2 is up-to-date
connect-jdbc_zookeeper_1_948bd8f572fb is up-to-date
Starting connect-jdbc_mysql_1_88681355e0c9 ...
Starting connect-jdbc_postgres_1_69535455d741 ...
connect-jdbc_kafka_1_8b8a8c2b124d is up-to-date
connect-jdbc_schema-registry_1_f5e9a2675ed1 is up-to-date
Recreating connect-jdbc_oracle_1_4be0ad4479f8 ... done
Starting connect-jdbc_mysql_1_88681355e0c9 ... done
Starting connect-jdbc_postgres_1_69535455d741 ... done
Attaching to connect-jdbc_oracle_1_4be0ad4479f8
oracle_1_4be0ad4479f8 | ORACLE PASSWORD FOR SYS, SYSTEM AND PDBADMIN: Admin123
oracle_1_4be0ad4479f8 | cp: error writing '/opt/oracle/dbca.rsp': No space left on device
oracle_1_4be0ad4479f8 | cp: failed to extend '/opt/oracle/dbca.rsp': No space left on device
oracle_1_4be0ad4479f8 | mv: cannot stat '/opt/oracle/product/12.2.0.1/dbhome_1/dbs/spfileORCLCDB.ora': No such file or directory
oracle_1_4be0ad4479f8 | mv: cannot stat '/opt/oracle/product/12.2.0.1/dbhome_1/dbs/orapwORCLCDB': No such file or directory
oracle_1_4be0ad4479f8 | mv: cannot stat '/opt/oracle/product/12.2.0.1/dbhome_1/network/admin/sqlnet.ora': No such file or directory
oracle_1_4be0ad4479f8 | mv: cannot stat '/opt/oracle/product/12.2.0.1/dbhome_1/network/admin/listener.ora': No such file or directory
oracle_1_4be0ad4479f8 | mv: cannot stat '/opt/oracle/product/12.2.0.1/dbhome_1/network/admin/tnsnames.ora': No such file or directory
oracle_1_4be0ad4479f8 | cp: error writing '/etc/oratab': No space left on device
oracle_1_4be0ad4479f8 | cp: failed to extend '/etc/oratab': No space left on device
oracle_1_4be0ad4479f8 | ORACLE_HOME = [/home/oracle] ? ORACLE_BASE environment variable is not being set since this
oracle_1_4be0ad4479f8 | information is not available for the current user ID .
oracle_1_4be0ad4479f8 | You can set ORACLE_BASE manually if it is required.
oracle_1_4be0ad4479f8 | Resetting ORACLE_BASE to its previous value or ORACLE_HOME
oracle_1_4be0ad4479f8 | The Oracle base remains unchanged with value /opt/oracle
oracle_1_4be0ad4479f8 | /opt/oracle/checkDBStatus.sh: line 27: cannot create temp file for here-document: No space left on device
oracle_1_4be0ad4479f8 | tail: cannot open '/opt/oracle/diag/rdbms/*/*/trace/alert*.log' for reading: No such file or directory
oracle_1_4be0ad4479f8 | tail: no files remaining
oracle_1_4be0ad4479f8 | #####################################
oracle_1_4be0ad4479f8 | ########### E R R O R ###############
oracle_1_4be0ad4479f8 | DATABASE SETUP WAS NOT SUCCESSFUL!
oracle_1_4be0ad4479f8 | Please check output for further info!
oracle_1_4be0ad4479f8 | ########### E R R O R ###############
oracle_1_4be0ad4479f8 | #####################################Argh! It was working just yesterday!
If you check out the space that Docker’s using, you’ll see it can be quite a hog
$ docker system df
TYPE TOTAL ACTIVE SIZE RECLAIMABLE
Images 86 7 36.87GB 35.87GB (97%)
Containers 13 0 76.66MB 76.66MB (100%)
Local Volumes 586 9 36.88GB 36.67GB (99%)
Build Cache 0 0 0B 0BThat’s over 70Gb of space!
If you load up the Docker UI you’ll see that it’s got a fixed size for the disk image that contains all the Docker data:
You could increase this size, or you could tidy up some of the unused content. You can prune these down easily, using the prune command. Go carefully though, because if you trash all your images then they’re going to have to be pulled back down from their repository or rebuilt, and that can take some time. Don’t do it just before you need to run the demo…
|
Note
|
You can do docker system prune -af but this trashes everything. I’d recommend taking a more structured approach to it, as shown below.
|
Pruning the unused volumes and stopped containers is easy:
$ docker volume prune
WARNING! This will remove all local volumes not used by at least one container.
Are you sure you want to continue? [y/N] y
Deleted Volumes:
dc502ff65c2dc5b51272834d7a6c58a36e8560876125b0b74f6024162fb28186
…
93d2e9fa57442a2f1a1837a7ecf14b069179bb814351e2888bdd0cfcd12d19dd
Total reclaimed space: 36.67GB$ docker container prune
WARNING! This will remove all stopped containers.
Are you sure you want to continue? [y/N] y
Deleted Containers:
…
bf8b43823b644c238b3d1e8722a6ad9383b2162a6d0a4d80bcc5c661c91ade32
Total reclaimed space: 76.66MBYou can do the same with images, but I don’t want to trash all my images, so I cheat and use a GUI - the excellent Portainer (which itself can be run under Docker 😃)
Now my Docker has a bit more space to breathe:
$ docker system df
TYPE TOTAL ACTIVE SIZE RECLAIMABLE
Images 33 1 17.2GB 17.15GB (99%)
Containers 1 1 0B 0B
Local Volumes 1 1 33.57kB 0B (0%)
Build Cache 0 0 0B 0BThis same problem can cause Elasticsearch to stop working also, if it runs out of space the index becomes read-only:
{"error":{"root_cause":[{"type":"cluster_block_exception","reason":"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"}],"type":"cluster_block_exception","reason":"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"},"status":403}