1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
|
headline: Tutorial
body:
- text: |
Twitter have a JSON API, so let's play with that. This URL gets
us the last 5 tweets about JSON:
- command: "curl 'http://search.twitter.com/search.json?q=json&rpp=5&include_entities=true'"
result: |
{"completed_in":0.108,"max_id":247677287108067328,"max_id_str":"247677287108067328","next_page":"?page=2&max_id=247677287108067328&q=json&rpp=5&include_entities=1","page":1,"query":"json","refresh_url":"?since_id=247677287108067328&q=json&include_entities=1","results":[{"created_at":"Mon, 17 Sep 2012 12:44:01 +0000","entities":{"hashtags":[],"urls":[{"url":"http:\/\/t.co\/XRvh1ZVw","expanded_url":"http:\/\/jase.im\/Ri7I0M","display_url":"jase.im\/Ri7I0M","indices":[112,132]}],"user_mentions":[{"screen_name":"imagemechanics","name":"Jason Cotterell","id":57271393,"id_str":"57271393","indices":[3,18]}]},"from_user":"_AaronNg","from_user_id":79771704,"from_user_id_str":"79771704","from_user_name":"NgChenChong","geo":null,"id":247677287108067328,"id_str":"247677287108067328","iso_language_code":"en","metadata":{"result_type":"recent"},"profile_image_url":"http:\/\/a0.twimg.com\/profile_images\/2523558403\/ek8mo4j4beq84iw28gjl_normal.jpeg","profile_image_url_https":"https:\/\/si0.twimg.com\/profile_images\/2523558403\/ek8mo4j4beq84iw28gjl_normal.jpeg","source":"<a href="http:\/\/twitter.com\/">web<\/a>","text":"RT @imagemechanics: iPhone 5 website teardown: How Apple compresses video using JPEG, JSON, and <canvas>.\nhttp:\/\/t.co\/XRvh1ZVw","to_user":null,"to_user_id":0,"to_user_id_str":"0","to_user_name":null}, ...
- text: |
There's lots of info and no whitespace, so to make it a bit more
legible we pipe it through jq, telling jq to just spit the input
back at us using the expression `.`:
- command: "curl 'http://search.twitter.com/search.json?q=json&rpp=5&include_entities=true' | jq '.'"
result: |
{
"since_id_str": "0",
"since_id": 0,
"results_per_page": 5,
"completed_in": 0.108,
"max_id": 247677287108067330,
"max_id_str": "247677287108067328",
"next_page": "?page=2&max_id=247677287108067328&q=json&rpp=5&include_entities=1",
"page": 1,
"query": "json",
"refresh_url": "?since_id=247677287108067328&q=json&include_entities=1",
"results": [
{
"from_user_name": "NgChenChong",
"from_user_id_str": "79771704",
"from_user_id": 79771704,
"from_user": "_AaronNg",
"iso_language_code": "en",
"text": "RT @imagemechanics: iPhone 5 website teardown: How Apple compresses video using JPEG, JSON, and <canvas>.\nhttp://t.co/XRvh1ZVw",
"to_user": null
/* lots more fields... */
},
/* lots more results... */
]
}
- text: |
Let's pull out the first tweet:
- command: "curl 'http://search.twitter.com/search.json?q=json&rpp=5&include_entities=true' | jq '.results[0]'"
result: |
{
"from_user_name": "NgChenChong",
"from_user_id_str": "79771704",
"from_user_id": 79771704,
"from_user": "_AaronNg",
"iso_language_code": "en",
"text": "RT @imagemechanics: iPhone 5 website teardown: How Apple compresses video using JPEG, JSON, and <canvas>.\nhttp://t.co/XRvh1ZVw",
"to_user": null
/* lots more fields... */
},
- text: |
For the rest of the examples, I'll leave out the `curl` command - it's not going to change.
There's still a lot of info we don't care about there, so we'll
restrict it down to the most interesting fields.
- command: "jq '.results[0] | {from_user, text}'"
result: |
{
"text": "RT @imagemechanics: iPhone 5 website teardown: How Apple compresses video using JPEG, JSON, and <canvas>.\nhttp://t.co/XRvh1ZVw",
"from_user": "_AaronNg"
}
- text: |
The `|` operator in jq feeds the output of one filter
(`.results[0]` which gets the first element of the results
array) into the input of another (`{from_user, text}` which
builds an object of those two fields).
Now let's get the rest of the tweets:
- command: "jq '.results[] | {from_user, text}'"
result: |
{
"text": "RT @imagemechanics: iPhone 5 website teardown: How Apple compresses video using JPEG, JSON, and <canvas>.\nhttp://t.co/XRvh1ZVw",
"from_user": "_AaronNg"
}
{
"text": "RT @_kud: iPhone 5 website teardown: How Apple compresses video using JPEG, JSON, and <canvas> -- http://t.co/Lhp92IqD",
"from_user": "blouerat"
}
{
"text": "Dynamic Forms Mobile App by pmadiset: Few Forms details are hosted on our server in the form of JSON. This JSON ... http://t.co/7KALdWaX",
"from_user": "cebu_iphone"
}
{
"text": "iPhone 5 website insanity - Video compressing using JPEG, JSON, and <canvas> http://t.co/28Jesbio (Oh #Apple, U So #Awesome...)",
"from_user": "dieswaytoofast"
}
{
"text": "RT @umutm: A very nice web-based JSON editor - http://t.co/M70snaIf",
"from_user": "Leolik"
}
- text: |
`.results[]` returns each element of the results array, one at a
time, which are all fed into `{from_user, text}`.
Data in jq is represented as streams of JSON values - every jq
expression runs for each value in its input stream, and can
produce any number of values to its output stream.
Streams are serialised by just separating JSON values with
whitespace. This is a `cat`-friendly format - you can just join
two JSON streams together and get a valid JSON stream.
If you want to get the output as a single array, you can tell jq to
"collect" all of the answers by wrapping the filter in square
brackets:
- command: "jq '[.results[] | {from_user, text}]'"
result: |
[
{
"text": "RT @imagemechanics: iPhone 5 website teardown: How Apple compresses video using JPEG, JSON, and <canvas>.\nhttp://t.co/XRvh1ZVw",
"from_user": "_AaronNg"
},
{
"text": "RT @_kud: iPhone 5 website teardown: How Apple compresses video using JPEG, JSON, and <canvas> -- http://t.co/Lhp92IqD",
"from_user": "blouerat"
},
{
"text": "Dynamic Forms Mobile App by pmadiset: Few Forms details are hosted on our server in the form of JSON. This JSON ... http://t.co/7KALdWaX",
"from_user": "cebu_iphone"
},
{
"text": "iPhone 5 website insanity - Video compressing using JPEG, JSON, and <canvas> http://t.co/28Jesbio (Oh #Apple, U So #Awesome...)",
"from_user": "dieswaytoofast"
},
{
"text": "RT @umutm: A very nice web-based JSON editor - http://t.co/M70snaIf",
"from_user": "Leolik"
}
]
- text: |
- - -
Next, let's try getting the URLs out of those API results as
well. In each tweet, the Twitter API includes a field called
"entities" which looks like this:
{
"user_mentions": [],
"urls": [{
"indices": [83, 103],
"display_url": "bit.ly/StniqT",
"expanded_url": "http://bit.ly/StniqT",
"url": "http://t.co/28Jesbio"
}],
"hashtags": [
{"indices": [108, 114], "text": "Apple" },
{"indices": [121, 129], "text": "Awesome"}
]
}
We want to pull out all of the "url" fields inside that array of url
objects, and make a simple list of strings to go along with the
"from_user" and "text" fields we already have.
- command: "jq '.results[] | {from_user, text, urls: [.entities.urls[].url]}'"
result: |
{
"urls": [
"http://t.co/XRvh1ZVw"
],
"text": "RT @imagemechanics: iPhone 5 website teardown: How Apple compresses video using JPEG, JSON, and <canvas>.\nhttp://t.co/XRvh1ZVw",
"from_user": "_AaronNg"
}
{
"urls": [
"http://t.co/Lhp92IqD"
],
"text": "RT @_kud: iPhone 5 website teardown: How Apple compresses video using JPEG, JSON, and <canvas> -- http://t.co/Lhp92IqD",
"from_user": "blouerat"
}
{
"urls": [
"http://t.co/7KALdWaX"
],
"text": "Dynamic Forms Mobile App by pmadiset: Few Forms details are hosted on our server in the form of JSON. This JSON ... http://t.co/7KALdWaX",
"from_user": "cebu_iphone"
}
{
"urls": [
"http://t.co/28Jesbio"
],
"text": "iPhone 5 website insanity - Video compressing using JPEG, JSON, and <canvas> http://t.co/28Jesbio (Oh #Apple, U So #Awesome...)",
"from_user": "dieswaytoofast"
}
{
"urls": [
"http://t.co/M70snaIf"
],
"text": "RT @umutm: A very nice web-based JSON editor - http://t.co/M70snaIf",
"from_user": "Leolik"
}
- text: |
Here we're making an object as before, but this time the urls
field is being set to `[.entities.urls[].url]`, which collects
all of the URLs defined in the entities object.
- text: |
- - -
Here endeth the tutorial! There's lots more to play with, go
read [the manual](/manual) if you're interested, and [download
jq](/download) if you haven't already.
|
