Liquid output agent (#1587)

* Start with a stripped copy of the data output agent.

* Run the data from the last event through a liquid template.

* Flatten the secret logic to allow for an easier switch to FormConfigurable.

* Switch to form configurable, and allow the content of the page to be configured.

* Allow the mime type to be changed.

* Cleanup.

* Write how this template works.

* Better default values.

* Cleanup.

* Refactor.

* Start testing.

* Test the validation.

* Test receive.

* Test the happy path through the receive web events.

* Test the authentication.

* This is actually a match.

* Refactor.

* Refactor.

* Refactor for better testing.

* Create a mode that lets the logic change. Start with a merge behavior.

* Refactor.

* Create a form configurable setting to change the mode.

* Document how the modes work.

* Wording change.

* Go with a singular secret.

* Fix typo.

* Fix the tests.

* Test cleanup.

* If not one of two types that receive events, ignore all received events.

* Set up these tests for the next set of changes.

* Isolate the method that gets the data for the liquid template.

* Look up past events to render through the liquid template.

* Implement a limit of 2 events.

* Extract a method.

* Hook the limit to options.

* Implement a limit of X events.

* Implement a date limit.

* Refactor the count limit.

* Limit by date with sql, not in-memory objects.

* This ordering is already built into the scope.

* Refactor the dates a bit.

* Put in a few checks around the date limits.

* Add the last X event options to the form and the documentation.

* Missed one bit of documentation.

* Add a view for a liquid output agent that makes it easy to retrieve the generated URL.

* This agent cannot accept events.

* Hardcode the possibilities instead of inspecting the integer.

* Do not be case sensitive on the date filter.

* Hardcode a limit of 5000, just in case no limit was provided.

* Better checks around the time period parsing.

* Test the hardcodes, and rename for consistency.

* Nevermind on that rename.

* Do not be case sensitive on this mode.

* Test that it works even when the casing on the mode is wrong.

* Here is more descriptive default content.

* Text change.

* The if is no longer necessary.

* Refactor.

* Move the limit down to 1000.

* Put a hard limit of 1000.

* Note the new event limit... limit.

* Validate for a valid event limit.

* Do not throw an error if someone types in a non-integer into this field.

* Text update.

* Typo.

* Add a link to the Liquid Templating engine.

app/models/agents/liquid_output_agent.rb

@@ -0,0 +1,214 @@
+module Agents
+  class LiquidOutputAgent < Agent
+    include WebRequestConcern
+    include FormConfigurable
+
+    cannot_be_scheduled!
+    cannot_create_events!
+
+    DATE_UNITS = %w[second seconds minute minutes hour hours day days week weeks month months year years]
+
+    description  do
+      <<-MD
+        The Liquid Output Agent outputs events through a Liquid template you provide.  Use it to create a HTML page, or a json feed, or anything else that can be rendered as a string from your stream of Huginn data. 
+
+        This Agent will output data at:
+
+        `https://#{ENV['DOMAIN']}#{Rails.application.routes.url_helpers.web_requests_path(agent_id: ':id', user_id: user_id, secret: ':secret', format: :any_extension)}`
+
+        where `:secret` is the secret specified in your options.  You can use any extension you wish.
+
+        Options:
+
+          * `secret` - A token that the requestor must provide for light-weight authentication.
+          * `expected_receive_period_in_days` - How often you expect data to be received by this Agent from other Agents.
+          * `content` - The content to display when someone requests this page.
+          * `mime_type` - The mime type to use when someone requests this page.
+          * `mode` - The behavior that determines what data is passed to the Liquid template.
+          * `event_limit` - A limit applied to the events passed to a template when in "Last X events" mode. Can be a count like "1", or an amount of time like "1 day" or "5 minutes".
+
+        # Liquid Templating
+
+        The content you provide will be run as a Liquid template. The data from the last event received will be used when processing the Liquid template.
+
+        To learn more about Liquid templates, go here: [http://liquidmarkup.org](http://liquidmarkup.org "Liquid Templating")
+
+        # Modes
+
+        ### Merge events
+
+          The data for incoming events will be merged. So if two events come in like this:
+
+```
+{ 'a' => 'b',  'c' => 'd'}
+{ 'a' => 'bb', 'e' => 'f'}
+```
+
+          The final result will be:
+
+```
+{ 'a' => 'bb', 'c' => 'd', 'e' => 'f'}
+```
+
+        This merged version will be passed to the Liquid template.
+
+        ### Last event in
+
+          The data from the last event will be passed to the template.
+
+        ### Last X events
+
+          All of the events received by this agent will be passed to the template
+          as the ```events``` array.
+
+          The number of events can be controlled via the ```event_limit``` option.
+          If ```event_limit``` is an integer X, the last X events will be passed
+          to the template.  If ```event_limit``` is an integer with a unit of
+          measure like "1 day" or "5 minutes" or "9 years", a date filter will
+          be applied to the events passed to the template.  If no ```event_limit```
+          is provided, then all of the events for the agent will be passed to
+          the template. 
+          
+          For performance, the maximum ```event_limit``` allowed is 1000.
+
+      MD
+    end
+
+    def default_options
+      content = <<EOF
+When you use the "Last event in" or "Merge events" option, you can use variables from the last event received, like this:
+
+Name: {{name}}
+Url:  {{url}}
+
+If you use the "Last X Events" mode, a set of events will be passed to your Liquid template.  You can use them like this:
+
+<table class="table">
+  {% for event in events %}
+    <tr>
+      <td>{{ event.title }}</td>
+      <td><a href="{{ event.url }}">Click here to see</a></td>
+    </tr>
+  {% endfor %}
+</table>
+EOF
+      {
+        "secret" => "a-secret-key",
+        "expected_receive_period_in_days" => 2,
+        "mime_type" => 'text/html',
+        "mode" => 'Last event in',
+        "event_limit" => '',
+        "content" => content,
+      }
+    end
+
+    form_configurable :secret
+    form_configurable :expected_receive_period_in_days
+    form_configurable :content, type: :text
+    form_configurable :mime_type
+    form_configurable :mode, type: :array, values: [ 'Last event in', 'Merge events', 'Last X events']
+    form_configurable :event_limit
+
+    def working?
+      last_receive_at && last_receive_at > options['expected_receive_period_in_days'].to_i.days.ago && !recent_error_logs?
+    end
+
+    def validate_options
+      if options['secret'].present?
+        case options['secret']
+        when %r{[/.]}
+          errors.add(:base, "secret may not contain a slash or dot")
+        when String
+        else
+          errors.add(:base, "secret must be a string")
+        end
+      else
+        errors.add(:base, "Please specify one secret for 'authenticating' incoming feed requests")
+      end
+
+      unless options['expected_receive_period_in_days'].present? && options['expected_receive_period_in_days'].to_i > 0
+        errors.add(:base, "Please provide 'expected_receive_period_in_days' to indicate how many days can pass before this Agent is considered to be not working")
+      end
+
+      if options['event_limit'].present?
+        if((Integer(options['event_limit']) rescue false) == false)
+          errors.add(:base, "Event limit must be an integer that is less than 1001.")
+        elsif (options['event_limit'].to_i > 1000)
+          errors.add(:base, "For performance reasons, you cannot have an event limit greater than 1000.")
+        end
+      else
+      end
+    end
+
+    def receive(incoming_events)
+      return unless ['merge events', 'last event in'].include?(mode)
+      memory['last_event'] ||= {}
+      incoming_events.each do |event|
+        case mode
+        when 'merge events'
+          memory['last_event'] = memory['last_event'].merge(event.payload)
+        else
+          memory['last_event'] = event.payload
+        end
+      end
+    end
+
+    def receive_web_request(params, method, format)
+      valid_authentication?(params) ? [liquified_content, 200, mime_type]
+                                    : [unauthorized_content(format), 401]
+    end
+
+    private
+
+    def mode
+      options['mode'].to_s.downcase
+    end
+
+    def unauthorized_content(format)
+      format =~ /json/ ? { error: "Not Authorized" }
+                       : "Not Authorized"
+    end
+
+    def valid_authentication?(params)
+      interpolated['secret'] == params['secret']
+    end
+
+    def mime_type
+      options['mime_type'].presence || 'text/html'
+    end
+
+    def liquified_content
+      template = Liquid::Template.parse(options['content'] || "")
+      template.render(data_for_liquid_template)
+    end
+
+    def data_for_liquid_template
+      case mode
+      when 'last x events'
+        events = received_events
+        events = events.where('events.created_at > ?', date_limit) if date_limit
+        events = events.limit count_limit
+        events = events.to_a.map { |x| x.payload }
+        { 'events' => events }
+      else
+        memory['last_event'] || {}
+      end
+    end
+
+    def count_limit
+      limit = Integer(options['event_limit']) rescue 1000
+      limit <= 1000 ? limit : 1000
+    end
+
+    def date_limit
+      return nil unless options['event_limit'].to_s.include?(' ')
+      value, unit = options['event_limit'].split(' ')
+      value = Integer(value) rescue nil
+      return nil unless value
+      unit = unit.to_s.downcase
+      return nil unless DATE_UNITS.include?(unit)
+      value.send(unit.to_sym).ago
+    end
+
+  end
+end

app/views/agents/agent_views/liquid_output_agent/_show.html.erb

@@ -0,0 +1,15 @@
+<p>
+  Data for this Agent is available at these URLs:
+</p>
+
+
+<ul>
+  <% url = lambda { |format| web_requests_url(:agent_id => @agent.id, :user_id => current_user.id, :secret => @agent.options['secret'], :format => format) } %>
+  <li><%= link_to url.call(:html), url.call(:html), :target => :blank %></li>
+  <li><%= link_to url.call(:json), url.call(:json), :target => :blank %></li>
+  <li><%= link_to url.call(:xml), url.call(:xml), :target => :blank %></li>
+</ul>
+
+<p>
+  ... or any other extension you wish, as the extension does not change the content or mime type.
+</p>

spec/models/agents/liquid_output_agent_spec.rb

@@ -0,0 +1,461 @@
+# encoding: utf-8
+
+require 'rails_helper'
+
+describe Agents::LiquidOutputAgent do
+  let(:agent) do
+    _agent = Agents::LiquidOutputAgent.new(:name => 'My Data Output Agent')
+    _agent.options = _agent.default_options.merge('secret' => 'secret1', 'events_to_show' => 3)
+    _agent.options['secret'] = "a secret"
+    _agent.user = users(:bob)
+    _agent.sources << agents(:bob_website_agent)
+    _agent.save!
+    _agent
+  end
+
+  describe "#working?" do
+    it "checks if events have been received within expected receive period" do
+      expect(agent).not_to be_working
+      Agents::LiquidOutputAgent.async_receive agent.id, [events(:bob_website_agent_event).id]
+      expect(agent.reload).to be_working
+      two_days_from_now = 2.days.from_now
+      stub(Time).now { two_days_from_now }
+      expect(agent.reload).not_to be_working
+    end
+  end
+
+  describe "validation" do
+    before do
+      expect(agent).to be_valid
+    end
+
+    it "should validate presence and length of secret" do
+      agent.options[:secret] = ""
+      expect(agent).not_to be_valid
+      agent.options[:secret] = "foo"
+      expect(agent).to be_valid
+      agent.options[:secret] = "foo/bar"
+      expect(agent).not_to be_valid
+      agent.options[:secret] = "foo.xml"
+      expect(agent).not_to be_valid
+      agent.options[:secret] = false
+      expect(agent).not_to be_valid
+      agent.options[:secret] = []
+      expect(agent).not_to be_valid
+      agent.options[:secret] = ["foo.xml"]
+      expect(agent).not_to be_valid
+      agent.options[:secret] = ["hello", true]
+      expect(agent).not_to be_valid
+      agent.options[:secret] = ["hello"]
+      expect(agent).not_to be_valid
+      agent.options[:secret] = ["hello", "world"]
+      expect(agent).not_to be_valid
+    end
+
+    it "should validate presence of expected_receive_period_in_days" do
+      agent.options[:expected_receive_period_in_days] = ""
+      expect(agent).not_to be_valid
+      agent.options[:expected_receive_period_in_days] = 0
+      expect(agent).not_to be_valid
+      agent.options[:expected_receive_period_in_days] = -1
+      expect(agent).not_to be_valid
+    end
+
+    it "should validate the event_limit" do
+      agent.options[:event_limit] = ""
+      expect(agent).to be_valid
+      agent.options[:event_limit] = "1"
+      expect(agent).to be_valid
+      agent.options[:event_limit] = "1001"
+      expect(agent).not_to be_valid
+      agent.options[:event_limit] = "10000"
+      expect(agent).not_to be_valid
+    end
+
+    it "should should not allow non-integer event limits" do
+      agent.options[:event_limit] = "abc1234"
+      expect(agent).not_to be_valid
+    end
+  end
+
+  describe "#receive?" do
+
+    let(:key)   { SecureRandom.uuid }
+    let(:value) { SecureRandom.uuid }
+
+    let(:incoming_events) do
+      last_payload = { key => value }
+      [Struct.new(:payload).new( { key => SecureRandom.uuid } ),
+       Struct.new(:payload).new( { key => SecureRandom.uuid } ),
+       Struct.new(:payload).new(last_payload)]
+    end
+
+    describe "and the mode is last event in" do
+
+      before { agent.options['mode'] = 'Last event in' }
+
+      it "stores the last event in memory" do
+        agent.receive incoming_events
+        expect(agent.memory['last_event'][key]).to equal(value)
+      end
+
+      describe "but the casing is wrong" do
+        before { agent.options['mode'] = 'LAST EVENT IN' }
+
+        it "stores the last event in memory" do
+          agent.receive incoming_events
+          expect(agent.memory['last_event'][key]).to equal(value)
+        end
+      end
+
+    end
+
+    describe "but the mode is merge" do
+
+      let(:second_key)   { SecureRandom.uuid }
+      let(:second_value) { SecureRandom.uuid }
+
+      before { agent.options['mode'] = 'Merge events' }
+
+      let(:incoming_events) do
+        last_payload = { key => value }
+        [Struct.new(:payload).new( { key => SecureRandom.uuid, second_key => second_value } ),
+         Struct.new(:payload).new(last_payload)]
+      end
+
+      it "should merge all of the events passed to it" do
+        agent.receive incoming_events
+        expect(agent.memory['last_event'][key]).to equal(value)
+        expect(agent.memory['last_event'][second_key]).to equal(second_value)
+      end
+
+      describe "but the casing on the mode is wrong" do
+
+        before { agent.options['mode'] = 'MERGE EVENTS' }
+
+        it "should merge all of the events passed to it" do
+          agent.receive incoming_events
+          expect(agent.memory['last_event'][key]).to equal(value)
+          expect(agent.memory['last_event'][second_key]).to equal(second_value)
+        end
+
+      end
+
+    end
+
+    describe "but the mode is anything else" do
+
+      before { agent.options['mode'] = SecureRandom.uuid }
+
+      let(:incoming_events) do
+        last_payload = { key => value }
+        [Struct.new(:payload).new(last_payload)]
+      end
+
+      it "should do nothing" do
+        agent.receive incoming_events
+        expect(agent.memory.keys.count).to equal(0)
+      end
+
+    end
+
+  end
+
+  describe "#count_limit" do
+    it "should have a default of 1000" do
+      agent.options['event_limit'] = nil
+      expect(agent.send(:count_limit)).to eq(1000)
+
+      agent.options['event_limit'] = ''
+      expect(agent.send(:count_limit)).to eq(1000)
+
+      agent.options['event_limit'] = '  '
+      expect(agent.send(:count_limit)).to eq(1000)
+    end
+
+    it "should convert string count limits to integers" do
+      agent.options['event_limit'] = '1'
+      expect(agent.send(:count_limit)).to eq(1)
+
+      agent.options['event_limit'] = '2'
+      expect(agent.send(:count_limit)).to eq(2)
+
+      agent.options['event_limit'] = 3
+      expect(agent.send(:count_limit)).to eq(3)
+    end
+
+    it "should default to 1000 with invalid values" do
+      agent.options['event_limit'] = SecureRandom.uuid
+      expect(agent.send(:count_limit)).to eq(1000)
+
+      agent.options['event_limit'] = 'John Galt'
+      expect(agent.send(:count_limit)).to eq(1000)
+    end
+
+    it "should not allow event limits above 1000" do
+      agent.options['event_limit'] = '1001'
+      expect(agent.send(:count_limit)).to eq(1000)
+
+      agent.options['event_limit'] = '5000'
+      expect(agent.send(:count_limit)).to eq(1000)
+    end
+  end
+
+  describe "#receive_web_request?" do
+
+    let(:secret) { SecureRandom.uuid }
+
+    let(:params) { { 'secret' => secret } }
+
+    let(:method) { nil }
+    let(:format) { nil }
+
+    let(:mime_type) { SecureRandom.uuid }
+    let(:content) { "The key is {{#{key}}}." }
+
+    let(:key)   { SecureRandom.uuid }
+    let(:value) { SecureRandom.uuid }
+
+    before do
+      agent.options['secret'] = secret
+      agent.options['mime_type'] = mime_type
+      agent.options['content'] = content
+      agent.memory['last_event'] = { key => value }
+      agents(:bob_website_agent).events.destroy_all
+    end
+
+    describe "and the mode is last event in" do
+
+      before { agent.options['mode'] = 'Last event in' }
+
+      it "should render the results as a liquid template from the last event in" do
+        result = agent.receive_web_request params, method, format
+
+        expect(result[0]).to eq("The key is #{value}.")
+        expect(result[1]).to eq(200)
+        expect(result[2]).to eq(mime_type)
+      end
+
+      describe "but the casing is wrong" do
+        before { agent.options['mode'] = 'last event in' }
+
+        it "should render the results as a liquid template from the last event in" do
+          result = agent.receive_web_request params, method, format
+
+          expect(result[0]).to eq("The key is #{value}.")
+          expect(result[1]).to eq(200)
+          expect(result[2]).to eq(mime_type)
+        end
+      end
+
+    end
+
+    describe "and the mode is merge events" do
+
+      before { agent.options['mode'] = 'Merge events' }
+
+      it "should render the results as a liquid template from the last event in" do
+        result = agent.receive_web_request params, method, format
+
+        expect(result[0]).to eq("The key is #{value}.")
+        expect(result[1]).to eq(200)
+        expect(result[2]).to eq(mime_type)
+      end
+
+    end
+
+    describe "and the mode is last X events" do
+
+      before do
+        agent.options['mode'] = 'Last X events'
+
+        agents(:bob_website_agent).create_event payload: {
+          "name" => "Dagny Taggart",
+          "book" => "Atlas Shrugged"
+        }
+        agents(:bob_website_agent).create_event payload: {
+          "name" => "John Galt",
+          "book" => "Atlas Shrugged"
+        }
+        agents(:bob_website_agent).create_event payload: {
+          "name" => "Howard Roark",
+          "book" => "The Fountainhead"
+        }
+
+        agent.options['content'] = <<EOF
+<table>
+  {% for event in events %}
+    <tr>
+      <td>{{ event.name }}</td>
+      <td>{{ event.book }}</td>
+    </tr>
+  {% endfor %}
+</table>
+EOF
+      end
+
+      it "should render the results as a liquid template from the last event in, limiting to 2" do
+        agent.options['event_limit'] = 2
+        result = agent.receive_web_request params, method, format
+
+        expect(result[0]).to eq <<EOF
+<table>
+  
+    <tr>
+      <td>Howard Roark</td>
+      <td>The Fountainhead</td>
+    </tr>
+  
+    <tr>
+      <td>John Galt</td>
+      <td>Atlas Shrugged</td>
+    </tr>
+  
+</table>
+EOF
+      end
+
+      it "should render the results as a liquid template from the last event in, limiting to 1" do
+        agent.options['event_limit'] = 1
+        result = agent.receive_web_request params, method, format
+
+        expect(result[0]).to eq <<EOF
+<table>
+  
+    <tr>
+      <td>Howard Roark</td>
+      <td>The Fountainhead</td>
+    </tr>
+  
+</table>
+EOF
+      end
+
+      it "should render the results as a liquid template from the last event in, allowing no limit" do
+        agent.options['event_limit'] = ''
+        result = agent.receive_web_request params, method, format
+
+        expect(result[0]).to eq <<EOF
+<table>
+  
+    <tr>
+      <td>Howard Roark</td>
+      <td>The Fountainhead</td>
+    </tr>
+  
+    <tr>
+      <td>John Galt</td>
+      <td>Atlas Shrugged</td>
+    </tr>
+  
+    <tr>
+      <td>Dagny Taggart</td>
+      <td>Atlas Shrugged</td>
+    </tr>
+  
+</table>
+EOF
+      end
+
+      it "should allow the limiting by time, as well" do
+
+        one_event = agent.received_events.select { |x| x.payload['name'] == 'John Galt' }.first
+        one_event.created_at = 2.days.ago
+        one_event.save!
+
+        agent.options['event_limit'] = '1 day'
+        result = agent.receive_web_request params, method, format
+
+        expect(result[0]).to eq <<EOF
+<table>
+  
+    <tr>
+      <td>Howard Roark</td>
+      <td>The Fountainhead</td>
+    </tr>
+  
+    <tr>
+      <td>Dagny Taggart</td>
+      <td>Atlas Shrugged</td>
+    </tr>
+  
+</table>
+EOF
+      end
+
+      it "should not be case sensitive when limiting on time" do
+
+        one_event = agent.received_events.select { |x| x.payload['name'] == 'John Galt' }.first
+        one_event.created_at = 2.days.ago
+        one_event.save!
+
+        agent.options['event_limit'] = '1 DaY'
+        result = agent.receive_web_request params, method, format
+
+        expect(result[0]).to eq <<EOF
+<table>
+  
+    <tr>
+      <td>Howard Roark</td>
+      <td>The Fountainhead</td>
+    </tr>
+  
+    <tr>
+      <td>Dagny Taggart</td>
+      <td>Atlas Shrugged</td>
+    </tr>
+  
+</table>
+EOF
+      end
+
+      it "it should continue to work when the event limit is wrong" do
+        agent.options['event_limit'] = 'five days'
+        result = agent.receive_web_request params, method, format
+
+        expect(result[0].include?("Howard Roark")).to eq(true)
+        expect(result[0].include?("Dagny Taggart")).to eq(true)
+        expect(result[0].include?("John Galt")).to eq(true)
+
+        agent.options['event_limit'] = '5 quibblequarks'
+        result = agent.receive_web_request params, method, format
+
+        expect(result[0].include?("Howard Roark")).to eq(true)
+        expect(result[0].include?("Dagny Taggart")).to eq(true)
+        expect(result[0].include?("John Galt")).to eq(true)
+      end
+
+      describe "but the mode was set to last X events with the wrong casing" do
+
+        before { agent.options['mode'] = 'LAST X EVENTS' }
+
+        it "should still work as last x events" do
+          result = agent.receive_web_request params, method, format
+          expect(result[0].include?("Howard Roark")).to eq(true)
+          expect(result[0].include?("Dagny Taggart")).to eq(true)
+          expect(result[0].include?("John Galt")).to eq(true)
+        end
+
+      end
+
+    end
+
+    describe "but the secret provided does not match" do
+      before { params['secret'] = SecureRandom.uuid }
+
+      it "should return a 401 response" do
+        result = agent.receive_web_request params, method, format
+
+        expect(result[0]).to eq("Not Authorized")
+        expect(result[1]).to eq(401)
+      end
+
+      it "should return a 401 json response if the format is json" do
+        result = agent.receive_web_request params, method, 'json'
+
+        expect(result[0][:error]).to eq("Not Authorized")
+        expect(result[1]).to eq(401)
+      end
+    end
+  end
+end