Jeremy Karn jeremykarn
jeremykarn
/ FromJsonInferSchema.java
Created
February 27, 2015 23:06
FromJsonInferSchema UDF from Mortar Pig 0.12 fork. Requires json.patch from https://issues.apache.org/jira/browse/PIG-1914
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| /* | |
| * Licensed to the Apache Software Foundation (ASF) under one | |
| * or more contributor license agreements. See the NOTICE file | |
| * distributed with this work for additional information | |
| * regarding copyright ownership. The ASF licenses this file | |
| * to you under the Apache License, Version 2.0 (the | |
| * "License"); you may not use this file except in compliance | |
| * with the License. You may obtain a copy of the License at | |
| * | |
| * http://www.apache.org/licenses/LICENSE-2.0 |
jeremykarn
/ example_mortar_iam_emr_policy.json
Last active
August 29, 2015 14:02
Example Mortar IAM EMR Policy document.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| { | |
| "Statement": [ | |
| { | |
| "Action": [ | |
| "s3:*" | |
| ], | |
| "Effect": "Allow", | |
| "Resource": [ | |
| "arn:aws:s3:::*elasticmapreduce", | |
| "arn:aws:s3:::*elasticmapreduce/*", |
jeremykarn
/ gist:d1c835b080ca56c8c7e3
Created
May 7, 2014 03:48
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| May 07 02:41:00 184.73.121.234 local7: 2014-05-07 02:41:00,389 [pool-2-thread-11] (RunJobTask.java:289) ERROR com.mortardata.mhc.model.RunJobTask - java.lang.RuntimeException: Unable to connect to Google to verify authentication. | |
| May 07 02:41:00 184.73.121.234 local7: 2014-05-07 02:41:00,428 [pool-2-thread-11] (MessageHandler.java:397) ERROR com.mortardata.mhc.MessageHandler - An unexpected error occurred while running your job. Please try running the job again, and contact us if you continue to experience issues. | |
| May 07 02:41:00 184.73.121.234 com.mortardata.mhc.exception.HawkInternalException: An unexpected error occurred while running your job. Please try running the job again, and contact us if you continue to experience issues. | |
| May 07 02:41:00 184.73.121.234 logger: <187> at com.mortardata.mhc.model.RunJobTask.handleException(RunJobTask.java:292) | |
| May 07 02:41:00 184.73.121.234 logger: <187> at com.mortardata.mhc.model.RunJobTask.doTask(RunJobTask.java:99) | |
| May 07 02:41:00 184.73.121.234 logge |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| { | |
| "Statement": [ | |
| { | |
| "Effect":"Allow", | |
| "Action":"dynamodb:*", | |
| "Resource":"*" | |
| } | |
| ] | |
| } |
jeremykarn
/ udf.pig
Created
September 30, 2013 13:57
Simple Pig example showing UDFs being called in each of the map and reduce phase.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| REGISTER ‘udf.py’ USING streaming_python AS my_udfs; | |
| tweets = LOAD 's3n://twitter-gardenhose-mortar/tweets' | |
| USING org.apache.pig.piggybank.storage.JsonLoader( | |
| 'text: chararray, place:tuple(name:chararray)'); | |
| -- my_length UDF is called in the mapper for each tweet. | |
| long_tweets = FILTER tweets BY my_udfs.my_length(text) > 50; |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| REGISTER ‘<python_file>’ USING streaming_python AS nltk_udfs; | |
| tweets = LOAD 's3n://twitter-gardenhose-mortar/tweets' | |
| USING org.apache.pig.piggybank.storage.JsonLoader( | |
| 'text: chararray, place:tuple(name:chararray)'); | |
| -- Group the tweets by place name and use a CPython UDF to find the top 5 bigrams | |
| -- for each of these places. | |
| bigrams_by_place = FOREACH (GROUP tweets BY place.name) GENERATE | |
| group AS place:chararray, |
jeremykarn
/ mongo pig schema generator
Last active
December 11, 2015 13:38
The Pig and Python scripts for a Mortar web project that generates the MongoLoader schema associated with a Mongo collection. Need to supply your own MongoDB connection details and your own s3 bucket.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| # | |
| # Copyright 2012 Mortar Data Inc. | |
| # | |
| # Licensed under the Apache License, Version 2.0 (the "License"); | |
| # you may not use this file except in compliance with the License. | |
| # You may obtain a copy of the License at | |
| # | |
| # http://www.apache.org/licenses/LICENSE-2.0 | |
| # | |
| # Unless required by applicable law or agreed to in writing, software |
jeremykarn
/ awsstatus.coffee
Last active
January 29, 2019 20:15
A Hubot script for monitoring the aws status rss feed and posting messages to a campfire room whenever there's a new update.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| # | |
| # Copyright 2012 Mortar Data Inc. | |
| # | |
| # Licensed under the Apache License, Version 2.0 (the "License"); | |
| # you may not use this file except in compliance with the License. | |
| # You may obtain a copy of the License at | |
| # | |
| # http://www.apache.org/licenses/LICENSE-2.0 | |
| # | |
| # Unless required by applicable law or agreed to in writing, software |