TESTING JAVASCRIPT… WITH RUBY?! :)

While Codest is primarily a Ruby shop, one of the many projects we are building is in JavaScript. It is a client-side library, which runs in a pretty challenging environment: it has to support pretty much every browser in existence, including very old ones, and to top it off it interacts with a gaggle of external scripts and services. It is tons of fun.

The curious case of non-bundled dependencies

With the above requirements comes a whole set of challenges not usually present in a client-side project, and one class of those issues has to do with testing. Of course, we have an impeccable set of unit tests and we’re running them against a very large matrix of browser / operating system combos in our CI/CD environment, but that alone does not explore everything that can go wrong.

Due to the overarching architecture of the ecosystem we’re running in, we depend on some external libraries being loaded alongside ours – that is to say, we do not bundle them with our code; we can’t and there’s nothing to be done about it. That presents an interesting challenge, because those libraries:

– might not even be there – if someone messes up implementing our library,

– might be there, but in wrong/incompatible versions,

– might have been modified by some other code that’s along for the ride in a particular implementation.

This shows clearly why unit tests are not enough: they test in isolation from the real world. Say we mock up some part of some external library’s public API, based on what we’ve discovered in it’s docs, and run an unit test against that. What does that prove?

You might be tempted to say “that means it works with the external library’s API”, but you’d be – sadly – wrong. It only means that it interacts correctly with a subset of the external library’s public API, and even then only with the version that we’ve mocked up.

What if the library literally changes out from under us? What if – out there in the wild – it gets some weird responses that make it hit a different, undocumented code path? Can we even protect against that?

Reasonable protection

Not 100%, no – the environment is too complex for that. But we can be reasonably sure that everything works how it’s supposed to with some generalized examples of what might happen to our code in the wild: we can do integration testing. The unit tests ensure that our code runs properly internally, and integration tests need to ensure that we “talk” properly with the libraries we cannot control. And not with stubs of them, either – actual, live libraries.

We could just use one of the available integration test frameworks for JavaScript, build a simple HTML page, throw some calls to our library and the remote libraries on it, and give it a good workout. However we don’t want to inundate any of the remote services’ endpoints with calls generated by our CI/CD environments. It would mess with some stats, possibly break some things, and – last but not least – we wouldn’t be very nice making someone’s production a part of our tests.

But was integration testing something so complex even possible? Since Ruby is our first and foremost love, we fell back on our expertise and started thinking about how we usually do integration testing with remote services in Ruby projects. We might use something like the vcr gem to record what is happening once, then keep replaying it to our tests whenever needed.

Enter proxy

Internally vcr achieves this by proxying requests. That was our a-ha! moment. We needed to proxy every request that shouldn’t hit anything on the “real” internet to some stubbed responses. Then that incoming data will get handed to the external library and our code runs as usual.

When prototyping something that seems complicated, we often fall back on Ruby as a time-saving method. We decided to make a prototype test harness for our JavaScript in Ruby to see how well the proxy idea will work before committing to building something more complicated in (possibly) JavaScript. It turned out to be surprisingly simple. In fact it’s so simple that we’re going to build one in this article together. 🙂

Lights, camera… wait, we forgot the props!

Of course we won’t be dealing with the “real thing” – explaining even a bit of what we’re building is far beyond the scope of a blog post. We can build something quick and easy to stand in for the libraries in question and then focus more on the Ruby part.

First, we need something to stand in for the external library we’re dealing with. We need it to exhibit a couple of behaviors: it should contact an external service, emit an event here and there, and most of all – not be built with easy integration in mind 🙂

Here’s what we’ll use:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
/* global XMLHttpRequest, Event */

const URL = 'https://poloniex.com/public/?command=returnTicker'
const METHOD = 'GET'

module.exports = {
  fetch: function () {
    var req = new XMLHttpRequest()
    req.responseType = 'json'
    req.open(METHOD, URL, true)
    var doneEvent = new Event('example:fetched')

    req.onreadystatechange = function (aEvt) {
      if (req.readyState === 4) {
        if (req.status === 200) {
          this.data = req.response
        } else {
          this.error = true
        }
        window.dispatchEvent(doneEvent)
      }
    }.bind(this)

    req.send(null)
  },
  error: false,
  data: {}
}

You’ll note that it calls an open API for some data – in this case some cryptocurrency exchange rates, since that’s what’s all the rage nowadays 😉 This API does not expose a sandbox and it is rate-limited, which makes it a prime example of something that should not be actually hit in tests.

You might notice that this is in fact an NPM-compatible module, while I’ve hinted at the script we normally deal with not being available on NPM for easy bundling. For this demonstration it’s enough that it exhibits certain behavior, and I’d rather have ease of explanation here at the cost of oversimplification.

Inviting the actors

Now we also need something to stand in for our library. Again, we’ll keep the requirements simple: it needs to call our “external” library and do something with the output. For the sake of keeping the “testable” part simple, we’ll also have it do dual logging: both to console, which is a bit harder to read in specs, and to a globally available array.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
window.remote = require('remote-calling-example')

window.failedMiserably = true
window.logs = []

function log (message) {
  window.logs.push(message)
  console.log(message)
}

window.addEventListener('example:fetched', function () {
  if (window.remote.error) {
    log('[EXAMPLE] Remote fetch failed')
    window.failedMiserably = true
  } else {
    log('[EXAMPLE] Remote fetch successful')
    log(`[EXAMPLE] BTC to ETH: ${window.remote.data.BTC_ETH.last}`)
  }
})

window.remote.fetch()

I’m also keeping the behavior staggeringly simple on purpose. As it is, there’s only two actually interesting code paths to spec for, so we won’t get swept under an avalanche of specs as we progress through the build.

It all just snaps together

We’ll throw a simple HTML page up:

1
2
3
4
5
6
7
8
<!DOCTYPE html>
<html>
<head>
  <title>Example page</title>
  <script type="text/javascript" src="./index.js"></script>
</head>
<body></body>
</html>

For the purposes of this demo we’ll bundle our HTML and JavaScript together with Parcel, a very simple web app bundler. I like Parcel a lot for times like these, when I’m throwing together a quick example or hacking on a back-of-napkin class idea. When you’re doing something so simple that configuring Webpack would take longer than writing the code you want, it’s the best.

It’s also unobtrusive enough that when I want to switch to something that’s a bit more battle-tested I don’t have to do almost any backpedaling from Parcel, which is not something you could say about Webpack. Note of caution, however – Parcel is in heavy development and issues can and will present themselves; I’ve had an issue where the transpiled JavaScript output was invalid on an older Node.js. Bottom line: don’t make it a part of your production pipeline just yet, but give it a spin nonetheless.

You can look at all the code we’re talking about on GitHub, by the way. The remote-calling package is here, and the package with the Ruby tests is here. Go look at it now if you want to see how we’re using Parcel, but if you’re here for the Ruby, careful – spoilers abound!

Harnessing the power of integration

Now we can construct our test harness.

For the spec framework itself, we’ve used rspec. In development environments we test using actual, non-headless Chrome – the task of running and controlling that has fallen to watir (and it’s trusty sidekick watir-rspec). For our proxy, we’ve invited Puffing Billy and rack to the party. Finally, we want to rerun our JavaScript build every time we run the specs, and that is achieved with cocaine.

That’s a whole bunch of moving parts, and so our spec helper is somewhat… involved even in this simple example. Let’s take a look at it and pick it apart.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
Dir['./spec/support/**/*.rb'].each { |f| require f }

TEST_LOGGER = Logger.new(STDOUT)

RSpec.configure do |config|
  config.before(:suite) { Cocaine::CommandLine.new('npm', 'run build', logger: TEST_LOGGER).run }

  config.include Watir::RSpec::Helper
  config.include Watir::RSpec::Matchers

  config.include ProxySupport

  config.order = :random
  BrowserSupport.configure(config)
end

Billy.configure do |c|
  c.cache = false
  c.cache_request_headers = false
  c.persist_cache = false
  c.record_stub_requests = true
  c.logger = Logger.new(File.expand_path('../log/billy.log', __FILE__))
end

Before the whole suite we’re running our custom build command through cocaine. That TEST_LOGGER constant might be a little bit of overkill, but we’re not very concerned about number of objects here. We’re of course running specs in random order, and we need to include all the goodies from watir-rspec. We also need to set up Billy so that it does no caching, but extensive logging to spec/log/billy.log. If you don’t know whether a request is actually getting stubbed or is hitting a live server (whoops!), this log is pure gold.

I’m sure that your keen eyes have already spotted ProxySupport and BrowserSupport. You might think that our custom goodies sit in there… and you’d be exactly right! Let’s see what BrowserSupport does first.

A browser, controlled

First, let’s introduce TempBrowser:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
class TempBrowser
  def get
    @browser ||= Watir::Browser.new(web_driver)
  end

  def kill
    @browser.close if @browser
    @browser = nil
  end

  private

  def web_driver
    Selenium::WebDriver.for(:chrome, options: options)
  end

  def options
    Selenium::WebDriver::Chrome::Options.new.tap do |options|
      options.add_argument '--auto-open-devtools-for-tabs'
      options.add_argument "--proxy-server=#{Billy.proxy.host}:#{Billy.proxy.port}"
    end
  end
end

Working backwards through the call tree, we can see that we’re setting up a Selenium browser options set for Chrome. One of the options we’re passing into it is instrumental in our setup: it instructs the Chrome instance to proxy everything through our Puffing Billy instance. The other option is just nice to have – every instance we run that’s not headless will have the inspection tools automatically open. That saves us uncountable amounts of Cmd+Alt+I’s per day 😉

After we set up the browser with these options, we pass it on to Watir and that’s pretty much it. The kill method is a bit of sugar which lets us repeatedly stop and restart the driver if we need to without throwing away the TempBrowser instance.

Now we can give our rspec examples a couple of superpowers. First of all, we get a nifty browser helper method which our specs will mostly revolve around. We can also avail ourselves of a handy method to restart the browser for a particular example if we’re doing something super-sensitive. Of course, we also want to kill the browser after the test suite is done, because under no circumstances do we want lingering Chrome instances – for the sake of our RAM.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
module BrowserSupport
  def self.browser
    @browser ||= TempBrowser.new
  end

  def self.configure(config)
    config.around(:each) do |example|
      BrowserSupport.browser.kill if example.metadata[:clean]
      @browser = BrowserSupport.browser.get
      @browser.cookies.clear
      @browser.driver.manage.timeouts.implicit_wait = 30
      example.run
    end

    config.after(:suite) do
      BrowserSupport.browser.kill
    end
  end
end

Wiring up the proxy

We’ve got a browser and spec helpers set up, and we’re ready to start proxying requests to our proxy. But wait, we didn’t set it up yet! We could bang out repeated calls to Billy for each and every example, but it’s better to get ourselves a couple of helper methods and save a couple thousand keystrokes. That’s what ProxySupport does.

The one we use in our test setup is slightly more complex, but here’s a general idea:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
# frozen_string_literal: true

require 'json'

module ProxySupport
  HEADERS = {
        'Access-Control-Allow-Methods' => 'GET',
        'Access-Control-Allow-Headers' => 'X-Requested-With, X-Prototype-Version, Content-Type',
        'Access-Control-Allow-Origin'  => '*'
      }.freeze

  def stub_json(url, file)
    Billy.proxy.stub(url).and_return({
      body: open(file).read,
      code: 200,
      headers: HEADERS.dup
    })
  end

  def stub_status(url, status)
    Billy.proxy.stub(url).and_return({
      body: '',
      code: status,
      headers: HEADERS.dup
    })
  end

  def stub_page(url, file)
    Billy.proxy.stub(url).and_return(
      body: open(file).read,
      content_type: 'text/html',
      code: 200
    )
  end

  def stub_js(url, file)
    Billy.proxy.stub(url).and_return(
      body: open(file).read,
      content_type: 'application/javascript',
      code: 200
    )
  end
end

We can stub:

  • HTML page requests – for our main “playground” page,
  • JS requests – to serve our bundled library,
  • JSON requests – to stub the request to the remote API,
  • and just a “whatever” request where we only care about returning a particular, non-200 HTTP response.

This will do nicely for our simple example. Speaking of examples – we should set up a couple!

Testing the good side

We need to wire together a couple of “routes” for our proxy first:

1
2
3
4
5
6
7
8
9
10
let(:page_url) { 'http://myfancypage.local/index.html' }
let(:js_url) { 'http://myfancypage.local/dist/remote-caller-example.js' }

let(:page_path) { './dist/index.html' }
let(:js_path) { './dist/remote-caller-example.js' }

before do
  stub_page page_url, page_path
  stub_js js_url, js_path
  end

It’s worth noting that from rspec’s perspective the relative paths here refer to the main project directory, so we’re loading our HTML and JS straight from the dist directory – as built by Parcel. You can already see how those stub_* helpers come in handy.

It’s also worth noting that we’re placing our “fake” website on a .local TLD. That way any runaway requests should not escape our local environment should something go awry. As a general practice I’d recommend at least not using “real” domain names in stubs unless absolutely necessary.

Another note we should make here is about not repeating ourselves. As the proxy routing gets more complex, with a lot more paths and URLs, there’ll be some real value in extracting this setup to a shared context and simply including it as needed.

Now we can spec out how our “good” path should look like:

1
2
3
4
5
6
7
8
9
10
11
12
13
context 'with correct response' do
  before do
    stub_json %r{http://poloniex.com/public(.*)}, './spec/fixtures/remote.json'
    goto page_url
    Watir::Wait.until { browser.execute_script('return window.logs.length === 2') }
  end

  it 'logs proper data' do
    expect(browser.execute_script('return window.logs')).to(
      eq(['[EXAMPLE] Remote fetch successful', '[EXAMPLE] BTC to ETH: 0.03619999'])
    )
  end
end

That’s pretty simple, isn’t it? Some more setup here – we stub the JSON response from the remote API with a fixture, go to our main URL and then… we wait.

The longest wait

The waits are a way of working around a limitation we’ve encountered with Watir – we can’t reliably wait for e.g. JavaScript events, so we need to cheat a bit and “wait” until the scripts have moved some object that we can access into a state that is of interest to us. The downside is that if that state never comes (due to a bug, for example) we need to wait for the watir waiter to time out. This drives the spec time up a bit. The spec still reliably fails though.

After the page has “stabilized” on the state that we’re interesting in, we can execute some more JavaScript in the context of the page. Here we call up the logs written to the public array and check whether they are what we expected.

As a side note – here’s where stubbing the remote request really shines. The response that gets logged to the console is dependent on the exchange rate returned by the remote API, so we couldn’t reliably test the log contents if they kept changing. There are ways of working around it of course but they aren’t very elegant.

Testing the bad branch

One more thing to test: the “failure” branch.

1
2
3
4
5
6
7
8
9
10
11
12
13
context 'with failed response' do
  before do
    stub_status %r{http://poloniex.com/public(.*)}, 404
    goto page_url
    Watir::Wait.until { browser.execute_script('return window.logs.length === 1') }
  end

  it 'logs failure' do
    expect(browser.execute_script('return window.logs')).to(
      eq(['[EXAMPLE] Remote fetch failed'])
    )
  end
end

It’s very similar to the above, with the difference being that we stub the response to return a 404 HTTP status code and expect a different log.

Let’s run our specs now.

1
2
3
4
5
6
7
8
9
10
11
12
% bundle exec rspec
Randomized with seed 63792
I, [2017-12-21T14:26:08.680953 #7303]  INFO -- : Command :: npm run build

Remote calling
  with correct response
    logs proper data
  with failed response
    logs failure

Finished in 23.56 seconds (files took 0.86547 seconds to load)
2 examples, 0 failures

Woohoo!

Conclusion

We’ve briefly discussed how JavaScript can be integration tested with Ruby. While originally it was considered more of a stopgap, we’re pretty happy with our little prototype now. We’re still considering a pure JavaScript solution, of course, but in the meantime we have a simple and practical way of reproducing and testing some very complex situations that we’ve encountered in the wild.

If you’re considering building something similar yourself, it should be noted it’s not without its limitations. For example if what you’re testing gets really AJAX-heavy, Puffing Billy will take a long time to respond. Also if you have to stub some SSL sources some more fiddling will be required – look into the watir documentation if it’s a requirement you have. We’ll surely keep exploring and looking for the best ways to deal with our unique use case – and we’ll make sure to let you know what we found out, too.

 

ELASTICSEARCH GOTCHAS – PART 1 :)

Elasticsearch is a search engine that is based on a trusted and mature library – Apache Lucene. Huge activity in git project repository and the implementation in such projects as GitHub, SoundCloud, Stack Overflow and LinkedIn bear testimony to its great popularity. The part “Elastic” says it all about the nature of the system, whose capabilities are enormous: from a simple file search on a small scale, through knowledge discovery, to big data analysis in real time.What makes Elastic a more powerful than the competition is the set of default configurations and behaviors, which allow to create a cluster and start adding documents to the index in a couple of minutes. Elastic will configure a cluster for you, will define an index and define the types of fields for the first document obtained, and when you add another server, it will automatically deal with dividing index data between servers.Unfortunately, the above mentioned automation makes it unclear to us what the default settings implicate, and it often turns out to be misleading. This article starts a series where I will be dealing with the most popular gotchas, which you might encounter during the process of Elastic-based app creation.

The number of shards cannot be changed

Let’s index the first document using index API:

1
2
3
4
5
$ curl -XPUT 'http://localhost:9200/myindex/employee/1' -d '{
    "first_name" :   "Jane",
    "last_name" :    "Smith",
    "steet_number":  12
  }'

In this moment, Elastic creates an index for us, titled myindex. What is not visible here is the number of shards assigned to the index. Shards can be understood as individual processes responsible for indexing, storing and searching of some part of documents of a whole index. During the process of document indexing, elastic decides in which shard a document should be found. That is based on the following formula:

shard = hash(document_id) % number_of_primary_shards

It is now clear that the number of primary shards cannot be changed for an index that contains documents. So, before indexing the first document, always create an index manually, giving the number of shards, which you think is sufficient for a volume of indexed data:

1
2
3
4
5
$ curl -XPUT 'http://localhost:9200/myindex/' -d '{
    "settings" : {
      "number_of_shards" : 10
    }
  }'

Default value for number_of_shards is 5.
This means that the index can be scaled to up to 5 servers, which collect data during indexation. For the production environment, the value of shards should be set depending on the expected frequency of indexation and the size of documents. For development and testing environments, I recommend setting the value to 1 – why so? It will be explained in the next paragraph of this article.

Sorting the text search results with a relatively small number of documents

When we search for a document with a phrase:

1
2
3
4
5
6
7
8
$ curl -XGET 'http://localhost:9200/myindex/my_type/_search' -d
  '{
    "query": {
      "match": {
        "title": "The quick brown fox"
      }
    }
  }'

Elastic processes text search in few steps, simply speaking:

  1. phrase from request is converted into the same identical form as the document was indexed in, in our case it will be set of terms:
    [“quick”, “brown”, “fox”] (“the” is removed because it’s insignificant),
  2. the index is being browsed to search the documents that contain at least one of the searched words,
  3. every document that is a match, is evaluated in terms of being relevant to the search phrase,
  4. the results are sorted by the calculated relevance and the first page of results is returned to the user.

In the third step, the following values (among others) are taken into account:

  1. how many words from the search phrase are in the document
  2. how often a given word occurs in a document (TF – term frequency)
  3. whether and how often the matching words occur in other documents (IDF – inverse document frequency) – the more popular the word in other documents, the less significant
  4. how long is the document

The functioning of IDF is important to us. Elastic for performance reasons does not calculate this value regarding every document in the index – instead, every shard (index worker) calculates its local IDF and uses it for sorting. Therefore, during the index search with low number of documents we may obtain substantially different results depending on the number of shards in an index and document distribution.

Let’s imagine that we have 2 shards in an index; in the first one there are 8 documents indexed with the word “fox”, and in the second one only 2 documents with the same word. As a result, the word “fox” will differ significantly in both shards, and this may produce incorrect results. Therefore, an index consisting of only one primary shard should be created for development purposes:

1
2
$ curl -XPUT 'http://localhost:9200/myindex/' -d
  '{"settings" : { "number_of_shards" : 1 } }'

Viewing the results of “far” search pages kills your cluster

As I’ve written before in previous paragraphs, documents in an index are shared between totally individual index processes – shards. Every process is completely independent and deals only with the documents, which are assigned to it.

When we search an index with millions of documents and wait to obtain top 10 results, every shard must return its 10 best-matched results to the cluster’s node, which initiated the search. Then the responses from every shard are joined together and the top 10 search results are chosen (within the whole index). Such approach allows to efficiently distribute the search process between many servers.

Let’s imagine that our app allows viewing 50 results per page, without the restrictions regarding the number of pages that can be viewed by a user. Remember that our index consists of 10 primary shards (1 per server).

Let’s see how the acquiring of search results will look like for the 1st and the 100th page:

Page No. 1 of search results:

  1. The node which receives a query (controller) passes it on to 10 shards.
  2. Every shard returns its 50 best matching documents sorted by relevance.
  3. After the responses has been received from every shard, the controller merges the results (500 documents).
  4. Our results are the top 50 documents from the previous step.

Page No. 100 of search results:

  1. The node which receives a query (controller) passes it on to 10 shards.
  2. Every shard returns its 5000 best matching documents sorted by relevance.
  3. After receiving responses from every shard, the controller merges the results (50000 documents).
  4. Our results are the documents from the previous step positioned 4901 – 5000.

Assuming that one document is 1KB in size, in the second case it means that ~50MB of data must be sent and processed around the cluster, in order to view 100 results for one user.

It’s not hard to notice, that network traffic and index load increases significantly with each successive result page. That’s why it is not recommended to make the “far” search pages available to the user. If our index is well configured, than the user should find the result he’s interested in on the first search pages, and we’ll protect ourselves from unnecessary load of our cluster. To prove this rule, check, up to what number of search result pages do the most popular web search engines allow viewing.

What’s also interesting is the observation of browser response time for successive search result pages. For example, below you can find response times for individual search result pages in Google Search (the search term was “search engine”):

1
2
3
4
5
6
7
| Search result page (10 documents per page) | Response time |
|--------------------------------------------|---------------|
| 1                                          | 250ms         |
| 10                                         | 290ms         |
| 20                                         | 350ms         |
| 30                                         | 380ms         |
| 38(last one available)                     |               |

In the next part, I will look closer into the problems regarding document indexing.