Skip to content

Instantly share code, notes, and snippets.

@DonRichards

DonRichards/Test_dataverse_linkage_examples.md

Last active April 7, 2025 19:58
Show Gist options
  • Save DonRichards/331f5ef990cb46f37ed6d55f5de5a040 to your computer and use it in GitHub Desktop.
Save DonRichards/331f5ef990cb46f37ed6d55f5de5a040 to your computer and use it in GitHub Desktop.
Download ZIP
test.
Raw
dataverse-download.js
// Dataverse download handler
console.log("Dataverse download script loaded");
document.addEventListener('DOMContentLoaded', function() {
console.log("DOM content loaded");
// The 2 values below are the only ones you need to change.
const DOMAIN = "https://archive.data.jhu.edu";
const DOI = "10.7281/T1WF3NZ7";
// Find all download links in tables - target links in the download column
const downloadLinks = document.querySelectorAll('table a[href="#"]');
console.log(`Found ${downloadLinks.length} download links`);
// Add click event listener to each link
downloadLinks.forEach(link => {
// Extract the filename (it's in a span inside the link)
const filenameSpan = link.querySelector('span');
const filenameText = filenameSpan ? filenameSpan.textContent.trim() : link.textContent.trim();
// Extract just the filename part without the file size in parentheses
const filename = filenameText.split(' ')[0];
console.log(`Processing link for file: ${filename}`);
// Add click event listener
link.addEventListener('click', async function(e) {
e.preventDefault();
// Show loading state
const originalHTML = link.innerHTML;
link.innerHTML = '<span style="color: rgb(0, 0, 0);">Loading...</span>';
link.style.pointerEvents = 'none';
try {
console.log(`Attempting to download: ${filename}`);
// Get file ID from Dataverse API
console.log(`Fetching dataset info from DOI: ${DOI}`);
const response = await fetch(`${DOMAIN}/api/datasets/:persistentId?persistentId=doi:${DOI}`);
const data = await response.json();
console.log("Dataset info retrieved:", data);
// Find the file with matching label
const file = data.data?.latestVersion?.files?.find(file => file.label === filename);
if (file) {
console.log(`Found file in dataset: ${JSON.stringify(file)}`);
const fileId = file.dataFile.id;
console.log(`File ID: ${fileId}`);
// Get direct download URL
const downloadUrl = `${DOMAIN}/api/access/datafile/${fileId}`;
console.log(`Direct download URL: ${downloadUrl}`);
// Create a temporary link to trigger the download
const downloadLink = document.createElement('a');
downloadLink.href = downloadUrl;
downloadLink.target = '_blank'; // Open in new tab to handle the redirect
downloadLink.download = filename;
document.body.appendChild(downloadLink);
downloadLink.click();
document.body.removeChild(downloadLink);
console.log(`Download initiated for ${filename}`);
} else {
console.error(`File "${filename}" not found in dataset`);
console.log("Available files:", data.data?.latestVersion?.files?.map(f => f.label) || []);
throw new Error(`File "${filename}" not found in dataset. Check the exact filename in the Dataverse repository.`);
}
} catch (error) {
console.error('Error downloading file:', error);
alert(`Error downloading ${filename}: ${error.message}`);
} finally {
// Restore original state
link.innerHTML = originalHTML;
link.style.pointerEvents = 'auto';
}
});
// Add visual indication that this is a download link
if (filenameSpan) {
// Add a subtle download arrow indicator
link.title = "Click to download";
link.style.cursor = "pointer";
}
});
});
Raw
index.html
<!DOCTYPE html>
<html data-bs-theme="light" lang="en">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0, shrink-to-fit=no">
<title>C3VD</title>
<link rel="stylesheet" href="assets/bootstrap/css/bootstrap.min.css">
<link rel="stylesheet" href="assets/css/before-after.compiled.css">
<link rel="stylesheet" href="assets/css/styles.css">
</head>
<body>
<div class="modal fade" role="dialog" tabindex="-1" id="video-modal" style="border-color: rgba(0,0,0,0);">
<div class="modal-dialog" role="document">
<div class="modal-content"><div>
<iframe id="iframeYoutube" width="560" height="315" src="https://www.youtube.com/embed/l-8SCV6-fUw?modestbranding=1&autohide=1&showinfo=0&controls=0&fs=0" frameborder="0"></iframe>
</div></div>
</div>
</div>
<div class="container" style="margin-top: 0px;">
<div class="row justify-content-center" style="margin-top: 15px;margin-bottom: 15px;">
<div class="col-9">
<h1 class="text-center">Colonoscopy 3D Video Dataset (C3VD)</h1>
<p class="text-center" style="margin: -10px;margin-bottom: 0px;"><em>from Colonoscopy 3D Video Dataset with Paired Depth from 2D-3D Registration</em><br></p>
<div class="row text-center justify-content-center">
<div class="col-4"><a href="https://scholar.google.com/citations?user=y8RsSRwAAAAJ&amp;hl=en">Taylor L. Bobrow</a></div>
<div class="col-4"><a href="https://scholar.google.com/citations?user=FjX3JAEAAAAJ">Mayank Golhar</a></div>
<div class="col-4"><a href="https://scholar.google.com/citations?hl=en&amp;user=YA_gt5EAAAAJ">Rohan Vijayan</a></div>
</div>
<div class="row text-center justify-content-center">
<div class="col-4"><a href="https://scholar.google.com/citations?hl=en&amp;user=a3eRS3sAAAAJ">Venkata S. Akshintala</a></div>
<div class="col-4"><a href="https://scholar.google.com/citations?user=AdiG_IAAAAAJ&amp;hl=en&amp;oi=sra">Juan R. Garcia</a></div>
<div class="col-4"><a href="https://scholar.google.com/citations?user=AdhZfSkAAAAJ&amp;hl=en">Nicholas J. Durr</a></div>
</div>
<div class="row">
<div class="col">
<p class="text-center" style="margin-bottom: 0px;">Johns Hopkins University</p>
</div>
</div>
</div>
</div>
<div class="row justify-content-center" style="margin-top: 15px;margin-bottom: 15px;">
<div class="col text-center"><img class="img-fluid" src="assets/img/sample.gif"></div>
</div>
<div class="row" style="margin-bottom: 15px;margin-top: 15px;">
<div class="col">
<h2 class="text-center">Abstract</h2>
<p>Screening colonoscopy is an important clinical application for several 3D computer vision techniques, including depth estimation, surface reconstruction, and missing region detection. However, the development, evaluation, and comparison of these techniques in real colonoscopy videos remain largely qualitative due to the difficulty of acquiring ground truth data. In this work, we present a <span style="text-decoration: underline;">C</span>olonoscopy <span style="text-decoration: underline;">3</span>D <span style="text-decoration: underline;">V</span>ideo <span style="text-decoration: underline;">D</span>ataset (C3VD) acquired with a high definition clinical colonoscope and high-fidelity colon models for benchmarking computer vision methods in colonoscopy. We introduce a novel multimodal 2D-3D registration technique to register optical video sequences with ground truth rendered views of a known 3D model. The different modalities are registered by transforming optical images to depth maps with a Generative Adversarial Network and aligning edge features with an evolutionary optimizer. This registration method achieves an average translation error of 0.321 millimeters and an average rotation error of 0.159 degrees in simulation experiments where error-free ground truth is available. The method also leverages video information, improving registration accuracy by 55.6% for translation and 60.4% for rotation compared to single frame registration. 22 short video sequences were registered to generate 10,015 total frames with paired ground truth depth, surface normals, optical flow, occlusion, six degree-of-freedom pose, coverage maps, and 3D models. The dataset also includes screening videos acquired by a gastroenterologist with paired ground truth pose and 3D surface models. The dataset and registration source code are available at durr.jhu.edu/C3VD.<br></p>
</div>
</div>
<div class="row" style="margin-bottom: 15px;margin-top: 15px;">
<div class="col-md-3">
<h2>Paper</h2><a href="https://arxiv.org/abs/2206.08903"><img src="assets/img/paper.png"></a>
</div>
<div class="col-md-9">
<h2><span style="color: rgb(51, 51, 51);">Citation</span></h2>
<p><span style="color: rgb(51, 51, 51);">Please cite our publication if you use code or data from this site.</span><br></p><pre style="background-color:#f0f0f0; padding:10px; border-radius:3px;">
<code class="language-latex"> @article{bobrow2023,
title={Colonoscopy 3D video dataset with paired depth from 2D-3D registration},
author={Bobrow, Taylor L and Golhar, Mayank and Vijayan, Rohan and Akshintala, Venkata S and Garcia, Juan R and Durr, Nicholas J},
journal={Medical Image Analysis},
pages={102956},
year={2023},
publisher={Elsevier},
}</code>
</pre>
</div>
</div>
<div class="row g-0 text-center align-items-center">
<div class="col-auto m-auto">
<h2>Results</h2>
<p style="margin-bottom: 4px;"><span style="color: rgb(13, 13, 13);">Colonoscopy video frames (left) are registered with rendered views of a ground truth 3D model (right). Edge features (overlay) are aligned by optimizing a loss function (bottom).</span><br></p><div>
<iframe allowfullscreen frameborder="0" src="https://www.youtube-nocookie.com/embed/9uJQxTSCyo0/?rel=0&modestbranding=1&autohide=1&showinfo=0" width="560" height="560"></iframe>
</div>
<p style="margin-top: 30px;margin-bottom: 4px;">Real colonoscope frames are paired with registered ground truth depth, surface normals, occlusion, and optical flow frames<br></p><div style="text-align: center; display: flex; justify-content: center;">
<div class='before-after-container'>
<div class='img background-img'></div>
<div class='img foreground-img'></div>
<input type="range" min="1" max="100" value="50" class="slider" name='slider' id="slider">
<div class='slider-button'></div>
<div class='btn-array'>
<button class = "btn depth focus">Depth</button>
<button class = "btn normals">Normals</button>
<button class = "btn occlusion">Occlusion</button>
<button class = "btn flow">Flow</button>
</div>
</div>
</div>
</div>
</div>
<div class="row" style="margin-top: 30px;">
<div class="col">
<h1 class="text-center">Dataset</h1>
<p>C3VD contains 22 <a href="#registered">registered videos</a> with paired ground truth depth, surface normals, optical flow, occlusion, six degree-of-freedom pose, coverage maps, and 3D models. The dataset also includes 4 <a href="#screening">screening colonoscopy videos</a> acquired by a gastroenterologist with paired ground truth pose and 3D surface models. <a href="#models">3D model files and molds</a> are also available for download. <a href="#code">Registration and rendering code</a> is made available on GitHub.</p>
</div>
</div>
<h2 class="text-center" id="registered-1" style="margin: 0px;margin-top: 20px;margin-bottom: 8px;">Registered Videos</h2>
<div class="row">
<div class="col">
<p style="margin-bottom: 0px;">For each registered video frame, the dataset includes:</p>
<ul>
<li><strong>Depth frame:&nbsp;</strong>depth along the camera frame's z-axis, clamped from 0-100 millimeters. Values are linearly scaled and encoded as a 16-bit grayscale image.<br></li>
<li><strong>Surface normal frame:&nbsp;</strong>reported with respect to the camera coordinate system. X/Y/Z components are stored in separate R/G/B color channels. Components are linearly scaled from ± 1 to 0-65535. Values are encoded as a 16-bit color image.</li>
<li><strong>Optical flow frame:&nbsp;</strong>computed flowing from the current frame to the previous frame, meaning the first frame in the sequence has no value. Values are saved in a color image, where the R-channel contains X-direction motion (left→right, -20 to 20 pixels), and the G-channel contains Y-direction motion (up→down, -20 to 20 pixels). Values are linearly scaled from 0 to 65535 and encoded as a 16-bit color image.<br></li>
</ul>
</div>
<div class="col">
<ul>
<li><strong>Occlusion frame:&nbsp;</strong>encoded as an 8-bit binary image. Pixels occluding other mesh faces within 100mm of the camera origin are assigned a value of 255, and all other pixels are assigned a value of 0.<br></li>
<li><strong>Camera pose:&nbsp;</strong>&nbsp;saved in a file named pose.txt. Each line contains a homogenous&nbsp;<span style="color: rgb(36, 41, 47);">camera-to-world transformation matrix</span>&nbsp;(flattened in row-major order) corresponding to each frame.</li>
</ul>
<p style="margin-bottom: 0px;">For each video sequence, we also provide:<br></p>
<ul>
<li><strong>3D model and coverage map:</strong> ground truth triangulated mesh, stored as a Wavefront OBJ file named coverage_mesh.obj. Coverage is embedded in the OBJ file by texture vertices assigned to each face (vt=1 is observed, vt=2 is unobserved).</li>
</ul>
</div>
</div>
<div class="row">
<div class="col">
<div class="table-responsive">
<table class="table table-sm">
<thead>
<tr>
<th>Model</th>
<th>Texture</th>
<th>Video</th>
<th># Frames</th>
<th></th>
<th>Download</th>
</tr>
</thead>
<tbody>
<tr>
<td>Cecum</td>
<td>1</td>
<td>a</td>
<td>276</td>
<td id="c1a"><span style="text-decoration: underline;">Preview</span></td>
<td><a href="#"><span style="color: rgb(0, 0, 0);">cecum_t1_a.zip</span></a>&nbsp;(2.86 GB)</td>
</tr>
<tr>
<td>Cecum</td>
<td>1</td>
<td>b</td>
<td>765</td>
<td id="c1b"><span style="text-decoration: underline;">Preview</span></td>
<td><a href="#"><span style="color: rgb(0, 0, 0);">cecum_t1_b.zip</span></a>&nbsp;(8.36 GB)</td>
</tr>
<tr>
<td>Cecum</td>
<td>2</td>
<td>a</td>
<td>370</td>
<td id="c2a"><span style="text-decoration: underline;">Preview</span></td>
<td><a href="#"><span style="color: rgb(0, 0, 0);">cecum_t2_a.zip</span></a>&nbsp;(3.71 GB)</td>
</tr>
<tr>
<td>Cecum</td>
<td>2</td>
<td>b</td>
<td>1,142</td>
<td id="c2b"><span style="text-decoration: underline;">Preview</span></td>
<td><a href="#"><span style="color: rgb(0, 0, 0);">cecum_t2_b.zip</span></a>&nbsp;(11.06 GB)</td>
</tr>
<tr>
<td>Cecum</td>
<td>2</td>
<td>c</td>
<td>595</td>
<td id="c2c"><span style="text-decoration: underline;">Preview</span></td>
<td><a href="#"><span style="color: rgb(0, 0, 0);">cecum_t2_c.zip</span></a>&nbsp;(6.13 GB)</td>
</tr>
<tr>
<td>Cecum</td>
<td>3</td>
<td>a</td>
<td>730</td>
<td id="c3a"><span style="text-decoration: underline;">Preview</span></td>
<td><a href="#"><span style="color: rgb(0, 0, 0);">cecum_t3_a.zip</span></a>&nbsp;(6.80 GB)</td>
</tr>
<tr>
<td>Cecum</td>
<td>4</td>
<td>a</td>
<td>465</td>
<td id="c4a"><span style="text-decoration: underline;">Preview</span></td>
<td><a href="#"><span style="color: rgb(0, 0, 0);">cecum_t4_a.zip</span></a>&nbsp;(5.04 GB)</td>
</tr>
<tr>
<td>Cecum</td>
<td>4</td>
<td>b</td>
<td>425</td>
<td id="c4b"><span style="text-decoration: underline;">Preview</span></td>
<td><a href="#"><span style="color: rgb(0, 0, 0);">cecum_t4_b.zip</span></a>&nbsp;(4.41 GB)</td>
</tr>
<tr>
<td>Descending Colon</td>
<td>4</td>
<td>a</td>
<td>148</td>
<td id="d4a"><span style="text-decoration: underline;">Preview</span></td>
<td><a href="#"><span style="color: rgb(0, 0, 0);">desc_t4_a.zip</span></a>&nbsp;(1.24 GB)</td>
</tr>
<tr>
<td>Sigmoid Colon</td>
<td>1</td>
<td>a</td>
<td>700</td>
<td id="s1a"><span style="text-decoration: underline;">Preview</span></td>
<td><a href="#"><span style="color: rgb(0, 0, 0);">sigmoid_t1_a.zip</span></a>&nbsp;(5.20 GB)</td>
</tr>
<tr>
<td>Sigmoid Colon</td>
<td>2</td>
<td>a</td>
<td>514</td>
<td id="s2a"><span style="text-decoration: underline;">Preview</span></td>
<td><a href="#"><span style="color: rgb(0, 0, 0);">sigmoid_t2_a.zip</span></a>&nbsp;(4.22 GB)</td>
</tr>
<tr>
<td>Sigmoid Colon</td>
<td>3</td>
<td>a</td>
<td>613</td>
<td id="s3a"><span style="text-decoration: underline;">Preview</span></td>
<td><a href="#"><span style="color: rgb(0, 0, 0);">sigmoid_t3_a.zip</span></a>&nbsp;(4.58 GB)</td>
</tr>
<tr>
<td>Sigmoid Colon</td>
<td>3</td>
<td>b</td>
<td>536</td>
<td id="s3b"><span style="text-decoration: underline;">Preview</span></td>
<td><a href="#"><span style="color: rgb(0, 0, 0);">sigmoid_t3_b.zip</span></a>&nbsp;(4.21 GB)</td>
</tr>
<tr>
<td>Transcending Colon</td>
<td>1</td>
<td>a</td>
<td>61</td>
<td id="t1a"><span style="text-decoration: underline;">Preview</span></td>
<td><a href="#"><span style="color: rgb(0, 0, 0);">trans_t1_a.zip</span></a>&nbsp;(0.59 GB)</td>
</tr>
<tr>
<td>Transcending Colon</td>
<td>1</td>
<td>b</td>
<td>700</td>
<td id="t1b"><span style="text-decoration: underline;">Preview</span></td>
<td><a href="#"><span style="color: rgb(0, 0, 0);">trans_t1_b.zip</span></a>&nbsp;(5.07 GB)</td>
</tr>
<tr>
<td>Transcending Colon</td>
<td>2</td>
<td>a</td>
<td>194</td>
<td id="t2a"><span style="text-decoration: underline;">Preview</span></td>
<td><a href="#"><span style="color: rgb(0, 0, 0);">trans_t2_a.zip</span></a>&nbsp;(1.58 GB)</td>
</tr>
<tr>
<td>Transcending Colon</td>
<td>2</td>
<td>b</td>
<td>103</td>
<td id="t2b"><span style="text-decoration: underline;">Preview</span></td>
<td><a href="#"><span style="color: rgb(0, 0, 0);">trans_t2_b.zip</span></a>&nbsp;(0.97 GB)</td>
</tr>
<tr>
<td>Transcending Colon</td>
<td>2</td>
<td>c</td>
<td>235</td>
<td id="t2c"><span style="text-decoration: underline;">Preview</span></td>
<td><a href="#"><span style="color: rgb(0, 0, 0);">trans_t2_c.zip</span></a>&nbsp;(1.83 GB)</td>
</tr>
<tr>
<td>Transcending Colon</td>
<td>3</td>
<td>a</td>
<td>250</td>
<td id="t3a"><span style="text-decoration: underline;">Preview</span></td>
<td><a href="#"><span style="color: rgb(0, 0, 0);">trans_t3_a.zip</span></a>&nbsp;(1.83 GB)</td>
</tr>
<tr>
<td>Transcending Colon</td>
<td>3</td>
<td>b</td>
<td>214</td>
<td id="t3b"><span style="text-decoration: underline;">Preview</span></td>
<td><a href="#"><span style="color: rgb(0, 0, 0);">trans_t3_b.zip</span></a>&nbsp;(1.66 GB)</td>
</tr>
<tr>
<td>Transcending Colon</td>
<td>4</td>
<td>a</td>
<td>382</td>
<td id="t4a"><span style="text-decoration: underline;">Preview</span></td>
<td><a href="#"><span style="color: rgb(0, 0, 0);">trans_t4_a.zip</span></a>&nbsp;(3.10 GB)</td>
</tr>
<tr>
<td>Transcending Colon</td>
<td>4</td>
<td>b</td>
<td>597</td>
<td id="t4b"><span style="text-decoration: underline;">Preview</span></td>
<td><a href="#"><span style="color: rgb(0, 0, 0);">trans_t4_b.zip</span></a>&nbsp;(4.61 GB)</td>
</tr>
</tbody>
</table>
</div>
</div>
</div>
<div class="row">
<div class="col">
<h2 class="text-center" id="screening" style="margin: 0px;margin-top: 20px;margin-bottom: 8px;">Screening Videos</h2>
<p>In addition to the video sequence, each file also contains camera pose information saved in a file named pose.txt. Each line contains a homogenous pose (flattened in row-major order) corresponding to each frame.</p>
</div>
</div>
<div class="row">
<div class="col">
<div class="table-responsive">
<table class="table table-sm">
<thead>
<tr>
<th>Model</th>
<th>Texture</th>
<th># Frames</th>
<th></th>
<th>Download</th>
</tr>
</thead>
<tbody>
<tr>
<td>Full Colon</td>
<td>1</td>
<td>5,458</td>
<td id="screen1"><span style="text-decoration: underline;">Preview</span></td>
<td><a href="#"><span style="color: rgb(0, 0, 0);">screening_t1.zip</span></a>&nbsp;(8.13 GB)</td>
</tr>
<tr>
<td>Full Colon</td>
<td>2</td>
<td>5,100</td>
<td id="screen2"><span style="text-decoration: underline;">Preview</span></td>
<td><a href="#"><span style="color: rgb(0, 0, 0);">screening_t2.zip</span></a>&nbsp;(7.09 GB)</td>
</tr>
<tr>
<td>Full Colon</td>
<td>3</td>
<td>4,726</td>
<td id="screen3"><span style="text-decoration: underline;">Preview</span></td>
<td><a href="#"><span style="color: rgb(0, 0, 0);">screening_t3.zip</span></a>&nbsp;(7.07 GB)</td>
</tr>
<tr>
<td>Full Colon</td>
<td>4</td>
<td>4,774</td>
<td id="screen4"><span style="text-decoration: underline;">Preview</span></td>
<td><a href="#"><span style="color: rgb(0, 0, 0);">screening_t4.zip</span></a>&nbsp;(7.36 GB)</td>
</tr>
</tbody>
</table>
</div>
</div>
</div>
<h2 class="text-center" id="models" style="margin: 0px;margin-top: 20px;margin-bottom: 8px;">3D Model Files</h2>
<div class="row">
<div class="col">
<div class="table-responsive">
<table class="table table-sm">
<thead>
<tr>
<th>Model</th>
<th>Object Download</th>
<th>Mold Download</th>
</tr>
</thead>
<tbody>
<tr>
<td>Ascending Colon</td>
<td><a href="#"><span style="color: rgb(0, 0, 0);">ascend_model.obj</span></a>&nbsp;(25.4 MB)</td>
<td><a href="#"><span style="color: rgb(0, 0, 0);">ascend_mold.zip</span></a>&nbsp;(18.7 MB)</td>
</tr>
<tr>
<td>Cecum</td>
<td><a href="#"><span style="color: rgb(0, 0, 0);">cecum_model.obj</span></a>&nbsp;(54.8 MB)</td>
<td><a href="#"><span style="color: rgb(0, 0, 0);">cecum_mold.zip</span></a>&nbsp;(24.9 MB)</td>
</tr>
<tr>
<td>Descending Colon</td>
<td><a href="#"><span style="color: rgb(0, 0, 0);">desc_model.obj</span></a>&nbsp;(38.0 MB)</td>
<td><a href="#"><span style="color: rgb(0, 0, 0);">desc_mold.zip</span></a>&nbsp;(26.6 MB)</td>
</tr>
<tr>
<td>Sigmoid Colon</td>
<td><a href="#"><span style="color: rgb(0, 0, 0);">sigmoid_model.obj</span></a>&nbsp;(20.8 MB)</td>
<td><a href="#"><span style="color: rgb(0, 0, 0);">sigmoid_mold.zip</span></a>&nbsp;(42.2 MB)</td>
</tr>
<tr>
<td>Transcending Colon</td>
<td><a href="#"><span style="color: rgb(0, 0, 0);">trans_model.obj</span></a>&nbsp;(18.3 MB)</td>
<td><a href="#"><span style="color: rgb(0, 0, 0);">trans_mold.zip</span></a>&nbsp;(24.1 MB)</td>
</tr>
<tr>
<td>Full Colon</td>
<td><a href="#"><span style="color: rgb(0, 0, 0);">full_model.obj</span></a>&nbsp;(194.8 MB)</td>
<td></td>
</tr>
</tbody>
</table>
</div>
</div>
</div>
<h2 class="text-center" style="margin: 0px;margin-top: 20px;margin-bottom: 8px;">Calibration Files</h2>
<div class="row justify-content-center">
<div class="col-auto">
<p><a href="http://drive.google.com/uc?export=view&amp;id=1ZTNyLx0p19U2Q3vl8dUe2YxxxA-9WisI&amp;confirm=t">cfhq190l_10x10mm_checkerboard_images.zip</a></p>
</div>
<div class="col-auto">
<p><a href="http://drive.google.com/uc?export=view&amp;id=1gUA7mAM7DSD9oCvPH1hgQ0s2hhg1kOaC&amp;confirm=t">cfhq190l_omnidirectional_params.zip</a></p>
</div>
</div>
<div class="row">
<div class="col">
<h2 class="text-center" id="screening-1" style="margin: 0px;margin-top: 20px;margin-bottom: 8px;"><span style="color: var(--bs-gray-800);">Revision History</span></h2>
<p><em>10/14/2023 |&nbsp;Updated the dataset file names to reflect peer-review completion.</em></p>
<p><em>05/03/2023 |&nbsp;Revised ground truth surface normal frames and updated naming convention:</em></p>
<ul>
<li>Corrected an error in the rendering code clipped negative surface normal z-components to 0 and resulted in some surface normals having a non-unitary length.</li>
<li>Surface normal axes were updated from +x pointing right, +y pointing up, and +z pointing out of the screen to +x pointing right, +y pointing down, and +z pointing into the screen to be consistent with the camera coordinate system as shown in Figure 3 of the paper.</li>
<li>The naming convention of the frames was updated to include zero padding (e.g. <em>0005_color.png</em>).</li>
</ul>
</div>
</div>
<div class="row justify-content-center">
<div class="col-auto text-center">
<h2 class="text-center" id="code" style="margin: 0px;margin-top: 20px;margin-bottom: 8px;"><a href="https://github.com/DurrLab/C3VD">Registration and Rendering Code Available on GitHub</a></h2><img src="assets/img/github.png">
</div>
</div>
<div class="row" style="margin: 0px;">
<div class="col text-center" style="margin: 0px;padding: 0px;margin-top: 40px;"><p xmlns:cc="http://creativecommons.org/ns#" >This work is licensed under <a href="http://creativecommons.org/licenses/by-nc-sa/4.0/?ref=chooser-v1" target="_blank" rel="license noopener noreferrer" style="display:inline-block;">CC BY-NC-SA 4.0<img style="height:22px!important;margin-left:3px;vertical-align:text-bottom;" src="https://mirrors.creativecommons.org/presskit/icons/cc.svg?ref=chooser-v1"><img style="height:22px!important;margin-left:3px;vertical-align:text-bottom;" src="https://mirrors.creativecommons.org/presskit/icons/by.svg?ref=chooser-v1"><img style="height:22px!important;margin-left:3px;vertical-align:text-bottom;" src="https://mirrors.creativecommons.org/presskit/icons/nc.svg?ref=chooser-v1"><img style="height:22px!important;margin-left:3px;vertical-align:text-bottom;" src="https://mirrors.creativecommons.org/presskit/icons/sa.svg?ref=chooser-v1"></a></p></div>
</div>
</div>
<script src="assets/js/jquery.min.js"></script>
<script src="assets/bootstrap/js/bootstrap.min.js"></script>
<script src="assets/js/before-after.js"></script>
<script src="assets/js/video-modal.js"></script>
<script src="assets/js/dataverse-download.js"></script>
</body>
</html>
Raw
test_dataverse.sh
#!/bin/bash
DOI="10.7281/T1WF3NZ7"
filename="VideoData.zip"
# Combine these
curl "https://archive.data.jhu.edu/api/datasets/:persistentId?persistentId=doi:$DOI" | jq ".data.latestVersion.files[] | select(.label==\"$filename\") | .dataFile.id" | while read -r file_id; do
echo "Downloading $filename"
download_url=$(curl -s -D - -o /dev/null "https://archive.data.jhu.edu/api/access/datafile/$file_id" | grep -i '^Location: ' | cut -d' ' -f2)
echo "Download URL: $download_url"
curl -L -o "$filename" "$(echo "$download_url" | tr -d '\r')"
done
# Check if the file was downloaded
if [ -f "$filename" ]; then
echo "File $filename downloaded successfully."
else
echo "Failed to download $filename."
fi
Raw
Test_dataverse_linkage_examples.md

TL;DR

To get the file list manually

curl "https://archive.data.jhu.edu/api/datasets/:persistentId?persistentId=doi:10.7281/T1WF3NZ7" | jq '.data.latestVersion.files[] | {label: .label, id: .dataFile.id}'

To down one of them

curl -s -D - -o /dev/null "https://archive.data.jhu.edu/api/access/datafile/14036" | grep -i '^Location: '

Dataverse Download Handler

A JavaScript implementation for seamlessly downloading files from a Dataverse repository directly from your website.

Overview

This system facilitates file downloads from a Dataverse data server by:

  1. Allowing users to click on file links in HTML tables
  2. Querying the Dataverse API to find the file ID
  3. Generating a direct download URL
  4. Initiating the browser download

File Structure

your-website/
├── index.html             # Your main HTML file with download tables
├── assets/
│   ├── js/
│   │   ├── dataverse-download.js   # The download handler script
│   │   └── ... (other JS files)
│   └── ... (other asset folders)
└── test_dataverse.sh      # Shell script for testing/reference (not used in git repo and is just for reference)

Installation

  1. Place dataverse-download.js in your website's assets/js/ directory
  2. Add the script tag to your HTML file, before the closing </body> tag:
<script src="assets/js/dataverse-download.js"></script>
  1. Ensure your HTML tables have links formatted like this: Use the href="#" to trigger the download and do not make this an actualy link to a file. The javascript will handle that.
<a href="#"><span style="color: rgb(0, 0, 0);">filename.zip</span></a>

How It Works

Download Logic

  1. Link Setup: When the page loads, the script finds all table links with href="#" and registers click event listeners.

  2. On Click: When a user clicks a download link:

    • The script extracts the filename from the link text
    • It queries the Dataverse API using the configured DOI
    • It searches for a file with a matching filename in the response
    • Once found, it gets the file ID
    • It constructs a direct download URL using this ID
    • It creates a temporary link element to initiate the download

  3. Time-Sensitive URLs: The implementation handles Dataverse's time-sensitive URLs correctly:

    • The direct API URL (https://archive.data.jhu.edu/api/access/datafile/${fileId}) is permanent
    • When accessed, this URL generates a fresh time-sensitive S3 URL and redirects to it
    • This ensures that even if the page is left open for hours, when a user clicks a link, they get a freshly generated URL with its full validity period (15 minutes)

Shell Script vs. JavaScript Implementation

The file test_dataverse.sh is a proof-of-concept shell script that demonstrates the same logic but for command-line use:

The JavaScript implementation follows the same logic but in a web-browser context.

Configuration

In dataverse-download.js, configure your dataset's DOI:

// DOI for the dataset
const DOI = "10.7281/T1WF3NZ7";

Replace this value with your dataset's DOI.

Test locally

python -m http.server 8000

Troubleshooting

File Not Found

If you see "File not found in dataset" errors:

  • Check the console log to see the list of available files, the filename is case sensitive.
  • Ensure the filename in your HTML matches exactly with the filename in Dataverse
  • Verify the DOI is correct

Download Not Starting

If the download doesn't start:

  • Check browser console for errors
  • Make sure pop-up blockers aren't preventing the download
  • Verify network connectivity to the Dataverse server
@mayankgolhar
Copy link

C3VD Website code: https://github.com/DurrLab/C3VD/tree/gh-pages
Relevant code for dowloading: https://github.com/DurrLab/C3VD/blob/6285dea2c0f4b6d9a3749180ffa3e4170db8c597/index.html#L140

@DonRichards
Copy link
Author

@mako-taco All set! This is a fully functional and tested JavaScript implementation. I’ve updated your index.html file to include the necessary script, which should trigger a download when clicked.

The setup instructions are in the README.md within this gist, but in short: you’ll add the JavaScript file to your project in the same way as your other scripts. Then, replace your existing index.html with the updated one.

Once you’ve entered your DOI into the JavaScript file and included all the files you want to add—those that have been uploaded to the Dataverse system—it should work as intended. The script targets all table rows and data cells () and will automatically fetch and download the files listed, assuming they’re formatted accordingly.

We can test this on the test server as soon as it’s uploaded—just make sure to update the domain and doi in the javascript file accordingly. I added a code comment to make it easier.

Let me know if you need any more help!

@mayankgolhar
Copy link

Thanks, Don, I really appreciate your help! I will work on setting up the new website and let you know if I have questions.

@mako-taco
Copy link

downloaded the virus, thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment