Open thumbnail logs in Grafana ↗ (internal; via the SSH SOCKS tunnel to hal49)

Failed thumbnail 2639:232167 2026-07-20 15:58:45 worker thumbnails-68db494c78-67tcv

timemachine_export · png 131 x 76 http://thumbnails-earthtime.cmucreatelab.org/thumbnail?bt=19841231&endDwell=0&et=19841231&format=png&fps=10&height=76&l=blsat&ps=0&root=https%3A%2F%2Fheadless.earthtime.org%23v=1128388%2C440233%2C1131876%2C442195%2Cpts&startDwell=0&t=0.03&tileFormat=mp4&width=131

Fatal error

Must specify boundsNWSE or boundsLTRB, unless fromScreenshot
/app/timemachine-thumbnail-server/thumbnail-server.rb:1171:in `compute_thumbnail'
/app/timemachine-thumbnail-server/thumbnail-server.rb:1624:in `serve_thumbnail'
/app/timemachine-thumbnail-server/thumbnail-server.rb:1749:in `<main>'

Last checkpoint

2026-07-20 15:58:45.293 THUMB 2639:232167:0 CHECKPOINTTHUMBNAIL FATALERROR {"REMOTE_ADDR":"10.1.100.5","HTTP_USER_AGENT":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130.0.0.0 Safari/537.36","HTTP_REFERER":"https://thumbnails-v2.createlab.org/status","codepath":"timemachine_export","FATALERROR":"Must specify boundsNWSE or boundsLTRB, unless fromScreenshot\n/app/timemachine-thumbnail-server/thumbnail-server.rb:1171:in `compute_thumbnail'\n/app/tim

Last line

2026-07-20 15:58:45.330 THUMB 2639:232167:0 request.body {"service":"Thumbnails","datetime":"2026-07-20T15:58:45+00:00","host":"-","level":"up","summary":"Last thumbnail failed","details":null,"payload":{},"valid_for_secs":null}

Referer: https://thumbnails-v2.createlab.org/status IP: 10.1.100.5 Browser: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130.0.0.0 Safari/537.36

Root

https://headless.earthtime.org#v=1128388,440233,1131876,442195,pts

Worker

thumbnails-68db494c78-67tcv

Raw stats

{"REMOTE_ADDR": "10.1.100.5", "HTTP_USER_AGENT": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130.0.0.0 Safari/537.36", "HTTP_REFERER": "https://thumbnails-v2.createlab.org/status", "codepath": "timemachine_export", "FATALERROR": "Must specify boundsNWSE or boundsLTRB, unless fromScreenshot\n/app/timemachine-thumbnail-server/thumbnail-server.rb:1171:in `compute_thumbnail'\n/app/timemachine-thumbnail-server/thumbnail-server.rb:1624:in `serve_thumbnail'\n/app/timemachine-thumbnail-server/thumbnail-server.rb:1749:in `<main>'"}

logs in Grafana ↗

Failed thumbnail 20461:118699 2026-07-20 15:40:38 worker thumbnails-68db494c78-gr657

timemachine_export · mp4 1280 x 720 http://thumbnails-v2.createlab.org/thumbnail?boundsLTRB=648,294,4170,2276&endDwell=0&endTime=145755&format=mp4&fps=6&height=720&labelsFromDataset&root=https://tiles.cmucreatelab.org/ecam/timemachines/accan1/2022-05-10.timemachine/&startDwell=0&startTime=145055&tileFormat=mp4&width=1280

Fatal error

404 Not Found
/usr/lib/ruby/3.2.0/open-uri.rb:369:in `open_http'
/usr/lib/ruby/3.2.0/open-uri.rb:760:in `buffer_open'
/usr/lib/ruby/3.2.0/open-uri.rb:214:in `block in open_loop'
/usr/lib/ruby/3.2.0/open-uri.rb:212:in `catch'
/usr/lib/ruby/3.2.0/open-uri.rb:212:in `open_loop'
/usr/lib/ruby/3.2.0/open-uri.rb:153:in `open_uri'
/usr/lib/ruby/3.2.0/open-uri.rb:740:in `open'
/usr/lib/ruby/3.2.0/open-uri.rb:29:in `open'
/app/timemachine-thumbnail-server/thumbnail-server.rb:429:in `start_thumbnail_not_screenshot'
/app/timemachine-thumbnail-server/thumbnail-server.rb:1182:in `compute_thumbnail'
/app/timemachine-thumbnail-server/thumbnail-server.rb:1624:in `serve_thumbnail'
/app/timemachine-thumbnail-server/thumbnail-server.rb:1749:in `<main>'

Last checkpoint

2026-07-20 15:40:38.769 THUMB 20461:118699:0 CHECKPOINTTHUMBNAIL FATALERROR {"REMOTE_ADDR":"10.1.100.5","HTTP_USER_AGENT":"Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Amazonbot/0.1; +https://developer.amazon.com/support/amazonbot) Chrome/119.0.6045.214 Safari/537.36","codepath":"timemachine_export","FATALERROR":"404 Not Found\n/usr/lib/ruby/3.2.0/open-uri.rb:369:in `open_http'\n/usr/lib/ruby/3.2.0/open-uri.rb:760:in `buffer_open'\n/usr/lib/ruby/3.2.0/open-uri.rb:214:in `block

Last line

2026-07-20 15:40:38.808 THUMB 20461:118699:0 request.body {"service":"Thumbnails","datetime":"2026-07-20T15:40:38+00:00","host":"-","level":"up","summary":"Last thumbnail failed","details":null,"payload":{},"valid_for_secs":null}

IP: 10.1.100.5 Browser: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Amazonbot/0.1; +https://developer.amazon.com/support/amazonbot) Chrome/119.0.6045.214 Safari/537.36

Root

https://tiles.cmucreatelab.org/ecam/timemachines/accan1/2022-05-10.timemachine/

Worker

thumbnails-68db494c78-gr657

Raw stats

{"REMOTE_ADDR": "10.1.100.5", "HTTP_USER_AGENT": "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Amazonbot/0.1; +https://developer.amazon.com/support/amazonbot) Chrome/119.0.6045.214 Safari/537.36", "codepath": "timemachine_export", "FATALERROR": "404 Not Found\n/usr/lib/ruby/3.2.0/open-uri.rb:369:in `open_http'\n/usr/lib/ruby/3.2.0/open-uri.rb:760:in `buffer_open'\n/usr/lib/ruby/3.2.0/open-uri.rb:214:in `block in open_loop'\n/usr/lib/ruby/3.2.0/open-uri.rb:212:in `catch'\n/usr/lib/ruby/3.2.0/open-uri.rb:212:in `open_loop'\n/usr/lib/ruby/3.2.0/open-uri.rb:153:in `open_uri'\n/usr/lib/ruby/3.2.0/open-uri.rb:740:in `open'\n/usr/lib/ruby/3.2.0/open-uri.rb:29:in `open'\n/app/timemachine-thumbnail-server/thumbnail-server.rb:429:in `start_thumbnail_not_screenshot'\n/app/timemachine-thumbnail-server/thumbnail-server.rb:1182:in `compute_thumbnail'\n/app/timemachine-thumbnail-server/thumbnail-server.rb:1624:in `serve_thumbnail'\n/app/timemachine-thumbnail-server/thumbnail-server.rb:1749:in `<main>'"}

logs in Grafana ↗

Failed thumbnail 13799:886157 2026-07-20 15:37:53 worker thumbnails-68db494c78-xfn84

timemachine_export · png 131 x 75 http://thumbnails-earthtime.cmucreatelab.org/thumbnail?bt=20141231&endDwell=0&et=20141231&format=png&fps=1&height=75&l=blsat%2Cextras_storyFriendlyDialog_davos_2024_australia&ps=0&root=https%3A%2F%2Fheadless.earthtime.org%23v=0%2C47651%2C2683438%2C1886871%2Cpts&startDwell=0&t=3&tileFormat=mp4&width=131

Fatal error

Must specify boundsNWSE or boundsLTRB, unless fromScreenshot
/app/timemachine-thumbnail-server/thumbnail-server.rb:1171:in `compute_thumbnail'
/app/timemachine-thumbnail-server/thumbnail-server.rb:1624:in `serve_thumbnail'
/app/timemachine-thumbnail-server/thumbnail-server.rb:1749:in `<main>'

Last checkpoint

2026-07-20 15:37:53.031 THUMB 13799:886157:0 CHECKPOINTTHUMBNAIL FATALERROR {"REMOTE_ADDR":"10.1.100.5","HTTP_USER_AGENT":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130.0.0.0 Safari/537.36","HTTP_REFERER":"https://thumbnails-v2.createlab.org/status","codepath":"timemachine_export","FATALERROR":"Must specify boundsNWSE or boundsLTRB, unless fromScreenshot\n/app/timemachine-thumbnail-server/thumbnail-server.rb:1171:in `compute_thumbnail'\n/app/ti

Last line

2026-07-20 15:37:53.061 THUMB 13799:886157:0 request.body {"service":"Thumbnails","datetime":"2026-07-20T15:37:53+00:00","host":"-","level":"up","summary":"Last thumbnail failed","details":null,"payload":{},"valid_for_secs":null}

Root

https://headless.earthtime.org#v=0,47651,2683438,1886871,pts

Worker

thumbnails-68db494c78-xfn84

Raw stats

{"REMOTE_ADDR": "10.1.100.5", "HTTP_USER_AGENT": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130.0.0.0 Safari/537.36", "HTTP_REFERER": "https://thumbnails-v2.createlab.org/status", "codepath": "timemachine_export", "FATALERROR": "Must specify boundsNWSE or boundsLTRB, unless fromScreenshot\n/app/timemachine-thumbnail-server/thumbnail-server.rb:1171:in `compute_thumbnail'\n/app/timemachine-thumbnail-server/thumbnail-server.rb:1624:in `serve_thumbnail'\n/app/timemachine-thumbnail-server/thumbnail-server.rb:1749:in `<main>'"}

logs in Grafana ↗

Failed thumbnail 10044:975672 2026-07-20 15:37:28 worker thumbnails-68db494c78-hlgqx

breathecam_export · mp4 1280 x 720 http://thumbnails-v2.createlab.org/thumbnail?endDwell=0&format=mp4&fps=6&fromScreenshot&height=720&minimalUI&root=https://breathecam.org%23v%3D483,892,4692,3259,pts%26t%3D1360.44%26ps%3D50%26bt%3D20260715174154%26et%3D20260715174445%26startDwell%3D0%26endDwell%3D0%26d%3D2026-07-15%26s%3Daccan2%26fps%3D6&startDwell=0&tileFormat=mp4&width=1280

Fatal error

exporter exited (1) before emitting header: Fetching https://tiles.cmucreatelab.org/ecam/timemachines/accan2/2026-07-15.timemachine/tm.json
Fetching https://tiles.cmucreatelab.org/ecam/timemachines/accan2/2026-07-15.timemachine/crf26-12fps-1424x800/r.json
TimeMachine has 4 levels and 28680 frames
Traceback (most recent call last):
  File "/app/timemachine-thumbnail-server/venv/bin/breathecam-thumbnail-export", line 6, in <module>
    sys.exit(main())
             ^^^^^^
  File "/app/timemachine-thumbnail-server/venv/lib/python3.12/site-packages/timemachine_video_export/breathecam_export_cli.py", line 109, in main
    start_frame = tm.frameno_from_date_after_or_equal(begin)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/timemachine-thumbnail-server/venv/lib/python3.12/site-packages/timemachine_video_export/timemachine.py", line 119, in frameno_from_date_after_or_equal
    for i, t in enumerate(self.capture_datetimes()):
                          ^^^^^^^^^^^^^^

Last checkpoint

2026-07-20 15:37:29.745 THUMB 10044:975672:0 CHECKPOINTTHUMBNAIL FATALERROR {"REMOTE_ADDR":"10.1.100.5","HTTP_USER_AGENT":"Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Amazonbot/0.1; +https://developer.amazon.com/support/amazonbot) Chrome/119.0.6045.214 Safari/537.36","codepath":"breathecam_export","FATALERROR":"exporter exited (1) before emitting header: Fetching https://tiles.cmucreatelab.org/ecam/timemachines/accan2/2026-07-15.timemachine/tm.json\nFetching https://tiles.cmuc

Last line

2026-07-20 15:37:29.782 THUMB 10044:975672:0 request.body {"service":"Thumbnails","datetime":"2026-07-20T15:37:29+00:00","host":"-","level":"up","summary":"Last thumbnail failed","details":null,"payload":{},"valid_for_secs":null}

IP: 10.1.100.5 Browser: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Amazonbot/0.1; +https://developer.amazon.com/support/amazonbot) Chrome/119.0.6045.214 Safari/537.36

Root

https://breathecam.org#v=483,892,4692,3259,pts&t=1360.44&ps=50&bt=20260715174154&et=20260715174445&startDwell=0&endDwell=0&d=2026-07-15&s=accan2&fps=6

Worker

thumbnails-68db494c78-hlgqx

Raw stats

{"REMOTE_ADDR": "10.1.100.5", "HTTP_USER_AGENT": "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Amazonbot/0.1; +https://developer.amazon.com/support/amazonbot) Chrome/119.0.6045.214 Safari/537.36", "codepath": "breathecam_export", "FATALERROR": "exporter exited (1) before emitting header: Fetching https://tiles.cmucreatelab.org/ecam/timemachines/accan2/2026-07-15.timemachine/tm.json\nFetching https://tiles.cmucreatelab.org/ecam/timemachines/accan2/2026-07-15.timemachine/crf26-12fps-1424x800/r.json\nTimeMachine has 4 levels and 28680 frames\nTraceback (most recent call last):\n  File \"/app/timemachine-thumbnail-server/venv/bin/breathecam-thumbnail-export\", line 6, in <module>\n    sys.exit(main())\n             ^^^^^^\n  File \"/app/timemachine-thumbnail-server/venv/lib/python3.12/site-packages/timemachine_video_export/breathecam_export_cli.py\", line 109, in main\n    start_frame = tm.frameno_from_date_after_or_equal(begin)\n                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/app/timemachine-thumbnail-server/venv/lib/python3.12/site-packages/timemachine_video_export/timemachine.py\", line 119, in frameno_from_date_after_or_equal\n    for i, t in enumerate(self.capture_datetimes()):\n                          ^^^^^^^^^^^^^^"}

logs in Grafana ↗

Failed thumbnail 13797:864779 2026-07-20 15:33:54 worker thumbnails-68db494c78-xfn84

timemachine_export · png 130 x 75 http://thumbnails-earthtime.cmucreatelab.org/thumbnail?bt=20200101&endDwell=0&et=20200101&format=png&fps=10&height=75&l=climate_week_2025_asset_id_solution_asset_id_solution_not_data_centers%2Cmb_labeled_monochrome_dark_mode%2Cclimate_week_2025_asset_id_solution_asset_id_solution_data_centers&ps=0&root=https%3A%2F%2Fspglobal.earthtime.org%23v=14.63751%2C-27.5815%2C0.53%2ClatLng&startDwell=0&t=0&tileFormat=mp4&width=130

Fatal error

Must specify boundsNWSE or boundsLTRB, unless fromScreenshot
/app/timemachine-thumbnail-server/thumbnail-server.rb:1171:in `compute_thumbnail'
/app/timemachine-thumbnail-server/thumbnail-server.rb:1624:in `serve_thumbnail'
/app/timemachine-thumbnail-server/thumbnail-server.rb:1749:in `<main>'

Last checkpoint

2026-07-20 15:33:54.090 THUMB 13797:864779:0 CHECKPOINTTHUMBNAIL FATALERROR {"REMOTE_ADDR":"10.1.100.5","HTTP_USER_AGENT":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130.0.0.0 Safari/537.36","HTTP_REFERER":"https://thumbnails-v2.createlab.org/status","codepath":"timemachine_export","FATALERROR":"Must specify boundsNWSE or boundsLTRB, unless fromScreenshot\n/app/timemachine-thumbnail-server/thumbnail-server.rb:1171:in `compute_thumbnail'\n/app/ti

Last line

2026-07-20 15:33:54.129 THUMB 13797:864779:0 request.body {"service":"Thumbnails","datetime":"2026-07-20T15:33:54+00:00","host":"-","level":"up","summary":"Last thumbnail failed","details":null,"payload":{},"valid_for_secs":null}

Root

https://spglobal.earthtime.org#v=14.63751,-27.5815,0.53,latLng

Worker

thumbnails-68db494c78-xfn84

Raw stats

{"REMOTE_ADDR": "10.1.100.5", "HTTP_USER_AGENT": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130.0.0.0 Safari/537.36", "HTTP_REFERER": "https://thumbnails-v2.createlab.org/status", "codepath": "timemachine_export", "FATALERROR": "Must specify boundsNWSE or boundsLTRB, unless fromScreenshot\n/app/timemachine-thumbnail-server/thumbnail-server.rb:1171:in `compute_thumbnail'\n/app/timemachine-thumbnail-server/thumbnail-server.rb:1624:in `serve_thumbnail'\n/app/timemachine-thumbnail-server/thumbnail-server.rb:1749:in `<main>'"}

logs in Grafana ↗

Finished thumbnail 11352:032791 2026-07-20 14:14:41 worker thumbnails-68db494c78-xfn84

screenshot · zip 540 x 540 x 180 frames

Finished

180 frames in 108.0 seconds (1.7 frames per second)
Frame efficiency 18.5%
IP: 10.1.100.5 Browser: Python-urllib/3.7

Root

https://headless.earthtime.org/#l=mb_labeled,smell_my_city_pgh_reports_top2026,plume_202607190400_202607200400&bt=20260719220000&et=20260720040000&v=40.42532,-79.91643,9.233,latLng&ps=2400&startDwell=0&endDwell=0

Worker

thumbnails-68db494c78-xfn84

Raw stats

{"REMOTE_ADDR": "10.1.100.5", "HTTP_USER_AGENT": "Python-urllib/3.7", "codepath": "screenshot", "headless_root": "https://headless.earthtime.org/#l=mb_labeled,smell_my_city_pgh_reports_top2026,plume_202607190400_202607200400&bt=20260719220000&et=20260720040000&v=40.42532,-79.91643,9.233,latLng&ps=2400&startDwell=0&endDwell=0&disableUI=true", "nshards": 5, "chromeRenderTimeSecs": 106.686576376, "videoFrameCount": 180, "frameEfficiency": 0.18480492813141683, "sizeBytes": 21055142, "totalTimeSecs": 107.961434314, "cpuTime": 51.125927}

logs in Grafana ↗