Skip to content

Instantly share code, notes, and snippets.

View FuhuXia's full-sized avatar

Fuhu Xia FuhuXia

  • REI Systems
  • Herndon, VA
  • 05:44 (UTC -04:00)
View GitHub Profile
@FuhuXia
FuhuXia / survey.py
Last active August 21, 2024 20:52
harvest source xml analysis
import requests
from time import sleep
CKAN_URL = 'https://catalog.data.gov'
# call api and get the list of harvest sources
harvest_source_url = f'{CKAN_URL}/api/action/package_search?fq=(dataset_type:harvest)&rows=1000'
response = requests.get(harvest_source_url)
harvest_sources = response.json()['result']['results']
@FuhuXia
FuhuXia / Smart-Card-Ubuntu.md
Last active June 21, 2025 22:09
How to get smart card reader c to work on Ubuntu 22.04

How to get smart card reader to work on Ubuntu 22.04

Hope this will save you hours of research on how to use your browser (Chrome/Chromium/Firefox) to load PIV card using Smart Card reader (SCR3310) to do Single Sign-On such as login.gov.

Make Ubuntu to read your PIV card

Step 1

$ sudo apt install libpam-sss
$ sudo apt install opensc-pkcs11
@FuhuXia
FuhuXia / TO-DO
Last active March 2, 2023 16:03
create github issue from slack message
1. set an initial timestamp, any message prior to it will be ignored, so that we dont have tons of issue created on the launch.
2. filter out certain messages, such as notification when a NewRelic issue is close.
3. auto close the issue if an issue-close NewRelic message or OK email are received.
4. remove the need fpr env variable GITHUB_USER.
5. apply lables to certain top priority messages such as prod server down notifications.
@FuhuXia
FuhuXia / solr-setup-for-catalog.sh
Last active April 22, 2022 15:47
solr-auth-setup.sh
#!/bin/bash
mkdir -p /var/solr/data/ckan
# add solr authentication
cat <<SOLRAUTH > /var/solr/data/security.json
{
"authentication":{
"blockUnknown": true,
"class":"solr.BasicAuthPlugin",
@FuhuXia
FuhuXia / 01-fcs-inventory-db-backup.sh
Last active December 11, 2021 01:36
Data.gov inventory migration script
#!/bin/bash
# Run this script on FCS inventory-harvester-xyz
set -o errexit
set -o pipefail
set -o nounset
AWS_DEFAULT_REGION=us-gov-west-1
service_name=inventory-db
@FuhuXia
FuhuXia / ckan_sync.sh
Created November 24, 2021 15:58
read ckan_ids.txt file and reindex each dataset id
#!/bin/sh
################
# USAGE: sh ckan_sync.sh > sync.log
################
tmpfile='./ckan_ids.txt'
while read id
do
@FuhuXia
FuhuXia / find-duplicate-ho-id.py
Last active November 24, 2021 16:00
Find CKAN records of data.json type with duplicate harvest_object_id
#!/usr/bin/env python
```
USAGE: python find-duplicate-ho-id.py > ids.txt
```
import urllib2
import json
import math
import sys