 Andrew Cantino
|
991e1466c6
fix spec when user agent is different
|
10 rokov pred |
 Akinori MUSHA
|
5031cbbbac
Migrate to RSpec's new expect syntax using Transpec.
|
10 rokov pred |
|
Akinori MUSHA
|
c21bada226
WebsiteAgent: Provide a variable _response_ for interpolation.
|
10 rokov pred |
|
Akinori MUSHA
|
df907c0290
Extend the spec to give a better idea on how to use `to_xpath`.
|
10 rokov pred |
|
Akinori MUSHA
|
0c490aa82d
Add a Liquid filter `to_xpath`, which quotes a string for use in XPath expression.
|
10 rokov pred |
|
Akinori MUSHA
|
863e2b8c70
WebsiteAgent should interpolate values from incoming event payload.
|
10 rokov pred |
|
Akinori MUSHA
|
fca8051e81
Add a parser type `text` to WebsiteAgent.
|
10 rokov pred |
|
Akinori MUSHA
|
7b6119f1f2
`"text": true` should have meant ".//text()", not "text()".
|
10 rokov pred |
|
Akinori MUSHA
|
a800342c29
WebsiteAgent: Add a spec for XPath returning an integer value.
|
10 rokov pred |
|
Akinori MUSHA
|
4d623c5893
WebsiteAgent: Introduce per-node XPath evaluation in extraction.
|
10 rokov pred |
|
Andrew Cantino
|
f4df522f2f
adding a basic RSS agent
|
10 rokov pred |
|
Andrew Cantino
|
fd2e8cd8b6
add headers option to WebsiteAgent
|
10 rokov pred |
|
Akinori MUSHA
|
e8751af629
Add a `user_agent` option to WebsiteAgent.
|
10 rokov pred |
|
Akinori MUSHA
|
85a7369e65
Use Faraday in WebsiteAgent and make HTTP backend library selectable.
|
10 rokov pred |
 Maximilian Clarke
|
9bf3c2c824
Updated WebsiteAgent to receive events
|
10 rokov pred |
|
Maximilian Clarke
|
19c005fe45
Modified website_agent to take an array of urls
|
10 rokov pred |
 Andrew Cantino
|
7d9279b871
Merge pull request #212 from knu/website_agent-force_encoding
|
10 rokov pred |
|
Akinori MUSHA
|
8ea2ba573f
Add :xpath support to WebsiteAgent.
|
11 rokov pred |
|
Akinori MUSHA
|
7bc20a0b44
Add :force_encoding support to WebsiteAgent.
|
11 rokov pred |
|
Andrew Cantino
|
99644a426d
Add XKCD hovertext to default seed and website agent
|
10 rokov pred |
|
Andrew Cantino
|
f4bae10250
minor code cleanup
|
11 rokov pred |
 Alex Piggott
|
b1898cc7ff
#154 Improvements to website deduplication logic
|
11 rokov pred |
|
Alex Piggott
|
7b38df61ed
#135 #141 2 deduplication fixes for the website agent
|
11 rokov pred |
 Albert Sun
|
7996954a3b
add a `basic_auth` option to the website agent
|
11 rokov pred |
|
Andrew Cantino
|
9c48338347
fix specs
|
11 rokov pred |
|
Andrew Cantino
|
00b7423dd7
add cached columns for event creation and last errors, reducing the number of SQL queries
|
11 rokov pred |
|
Albert Sun
|
43194c3c1b
in website agent with type json, allow extract to be blank; in which case, the entire json object will be stored as the payload
|
11 rokov pred |
|
Andrew Cantino
|
7372244d0f
return false from working? when an agent's most recent log is an error
|
11 rokov pred |
|
Andrew Cantino
|
00727fbd4d
add Agent Logs; add logging to WebsiteAgent; refactor flash notices and add event notices
|
11 rokov pred |
 itkevin
|
fd8761177f
When crawling websites tith relative URLs, make them absolute
|
11 rokov pred |
