API issues

Discussion in 'Building with the Kayako API' started by Phil R, Aug 15, 2016.

  1. Phil R

    Phil R Established Member

    Given the response that it is unlikely a Python SDK will be available, I have been exploring what it will take to write one.

    With a little over four hours work, I can retrieve from 'tests' for all, one including accepting params.

    Posting is unlikely to take much effort from here, but I am finding some areas difficult.

    Does flat mode work?

    From my testing, no, it doesn't.

    Examples
    GET /api/v1/cases/2/notes?_flat=true
    GET /api/v1/cases/2/notes/4?_flat=true

    Both are valid requests, but neither return expected results.

    The first does not even return x['resources'], but whilst the second does return it, it is empty.

    The notes contains a 'user' resource, so according to the API description, the value should be populated to the x['resources'].

    The same applies if I look at /api/v1/users/1 or /api/v1/users as an example.

    Does flat mode really save space?

    Given the above, I cannot manually examine the theory. However, on paper, it should really consume more resources?

    If I retrieve 5 users in _flat mode, then not just have I received a list of those users and references to sub-resources, such as their team-membership. But I now also have x['resources'] block will all said resource content, which increases the response size.

    Yes, the resource content should be de-duplicated and if working, will stop me needed to do the two following things.

    1: Implement sub-model retrieval
    2: Implement model caching such as memcached

    Afterall, _flat seems to indicate that it offer "live" resources, so if a sub-resource changed, the x['resources'] now contains fresh data, without needing to make a second round trip.

    In addition to the data size, execution time seems to be prolonged server side.

    In a 'users' retrieval, I am no longer just getting a ref to a sub-resource. I am also now waiting for the server to obtain the sub-resource data as well. Database lookup time is obviously increased.

    In some API endpoints, that is a lot of database lookups and retrieval to occur in a single API call.

    Incomplete Tests API?

    The documentation is to be fair, lacking a little, but not too much. The problem is, it lacks in a very technical area and I have found I need to go through trial and error just to get what I need.

    Seriously, I have not looked at other endpoints yet, but if Tests is anything to go by, I worry. This could also impact design decisions in my SDK.

    To name a few, none of the Tests (single, all) retrieval returns 'user' or 'is_odd'.

    Worse, some that are implemented are returning data types that they really shouldn't. Examples are 'is_boolean', which should only ever return true/false but returns null, plus 'option' should should only ever return a value of four, none of which is null.

    I note the _empty option that is available from special options, which does take care of this somewhat. It at least allows me to detect unset values, versus null values (null is a valid JSON value and there is nothing in the Kayako API that prohibits null being an actual value).

    This leaves a confusion issue, why is 'is_boolean' present when null, but 'is_odd' not present when null?

    Why am I missing 'user' completely, when it's clear that this value is require in the sense of delivering test cases for the SDK (I cannot reliably test sub-resource lazy loading).

    A proper top level schema?

    Can we get a proper top level schema please?

    That is, under the root {} response, what are the immediate attributed available?

    I ask, as I am finding new ones all the time and they are more than often not easy to find out what they are for.

    If you look at Users, Cases, Notes and any other, the attributes for each item that can be found under x['data'] is very well defined (with exception of the above).

    However, the root is rather well undefined, or at least centrally.

    The Responses section tells us plenty about 'status, 'data', 'resource', 'total_count'. Then 'errors', 'notifications', 'logs' are not quite as easy to spot, even though they are just below.

    There are additional items in Authentication, Paging.

    When I was trying to write a model loader, I hit validation errors, because I was encountering elements that didn't exist according to the docs. Or at least didn't exist if you didn't read areas that outside of the Response portion in which I should really have found it.

    CSRF Protection, the point?

    Forgive me if I am having problems with this. What does the Kayako implementation of the CSRF protection offer that is not already present?

    I can see the logic of having an 'enforced' mode in which the token MUST be present.

    However, the API already operates on the idea of Sessions. The CSRF token itself according to the wording of the docs, it even tied to the session.

    So why is the mechanism separate?

    i.e. why not have just made sessions enforced. No session token, no access.

    Right now, CSRF is just a secondary item of information as I MUST supply a session anyway for my CSRF token to be checked against.

    Of real importance here, is that the CSRF token is only needed for unsafe methods, which also means that it is not an overall protection that it implies.

    Session importance

    The Authentication section calls out the importance of sessions, in that some functions cannot be used.

    I have written in session management to my SDK anyway, because this will be critical to being able to use the API using OTP (otherwise I would be asking or a new token every few seconds, which is not very good for autonomous operations).

    It doesn't tell me anything about any other function that is reliant on the session to operate.

    Other portions of the docs do not call out when a sessionless operation will fail.
     
  2. Paul Kuruvilla

    Paul Kuruvilla Staff Member

    Re: Does flat mode work?

    Flat mode only applies to nested resources. The purpose of flat mode is to save space when we’re sending multiple nested resources (that could be duplicates) down the wire. In the examples that you provided none of the nested resources are included because we disable nested resources by default.

    Here is a live example of flat mode working: (it is the same as your exact example, just that this time - I included the user resource by adding ?include=user in the query string)

    With flat mode on: (GET /api/v1/cases/3/notes.json?include=user&_flat=true)
    flat_mode_on.png


    With flat mode off: (GET /api/v1/cases/3/notes.json?include=user)
    flat_mode_off.png



    The resources node should always be present (but might be empty) if flat mode is enabled, irrespective of whether it is a bulk call or a call for a single item.

    Re: Does flat mode really save space?
    All items in the resources block will not be present in their expanded form in the primary data node - they will only be a reference (resource_type and id ). i.e. In flat mode - every resource will only appear once in its expanded form. Everywhere else, it will be a reference.

    The benefit that you are talking about is achieved using the include parameter (You can read more about this at https://developer.kayako.com/api/v1/reference/partial_output/). If you include the nested resources that you know you are going to need, that should save you the second round trip. Note: All nested resources are not included by default - they have to be manually included by using the include parameter.

    Whether flat mode is being used or not should affect the need for a second round trip - the only purpose for flat mode is to avoid sending duplicate nested resources over the wire.

    I hope the confusion regarding this is cleared up - this only happens when you explicitly request for nested resources to be included (we don’t include any by default). For quick and speedy responses from the API, make sure to only include nested resources if you absolutely need them :)

    Re: Incomplete Tests API?

    The docs and implementation are definitely outdated, the dev team is currently working on this.

    Re: A proper top level schema?

    Noted, this can be improved.

    Re: CSRF Protection, the point?

    The main difference between a session ID and a CSRF token is that the first is set as a cookie, and the second is not (and hence is not sent along with requests originating from other sites). This is important for web-based API clients (like our agent interface), explained in-depth in http://www.redotheweb.com/2015/11/09/api-security.html.

    Re: Session importance
    Hmm, I’m not sure on this one.. Will post back here once I have an answer
     
    Last edited: Aug 18, 2016
  3. Phil R

    Phil R Established Member

    Many thanks for the reply.

    Regarding flat mode. Your examples have assisted in this, but have also highlighted the deficiency in the documentation that resulted in my confusion.

    You have provided a "include=" parameter, which is completely omitted from the flat mode docs, as it only assists this include.

    https://developer.kayako.com/api/v1/reference/special_options/#flat-mode-3

    Regarding space saving. Yes, now I know that "include" is where _flat comes into it, I can clearly see the space saving.

    CSRF

    Your response does not stack up.

    CSRF is implemented over the X-CSRF-Token header.

    The sessions do not use cookies either. they use the X-Session-ID header (or _session_id parameter).

    I understand the logic behind normal cookies and the inherent risks, along with the goal of using a CSRF token to protect this. However, this is also based on situations in which you are using a standard cookie implementation, which you are not doing.

    Am I missing another undocumented trick, such as the ability to send a "Cookie: _session_id=" header?
     
  4. Phil R

    Phil R Established Member

    Just to add to my reply regarding the undocumented changes to the "tests" endpoint.

    Given that the nested "user" resource is no longer attached, you need to provide some other form of nested resource for test purposes.

    Oddly enough, the nested resource element is something I am struggling with when working on my SDK, so some form of predictable nesting would benefit.

    Otherwise, full unit and functional testing also cannot be added to any SDK.
     
  5. Paul Kuruvilla

    Paul Kuruvilla Staff Member

    Hi Phil,

    Yep, you guessed right :) We do support cookie based auth for our web clients, you can see this in action by signing into a Kayako instance and hitting an authenticated API URL in your browser.

    We're planning on retaining the `user` resource for that very purpose - this is one in the list of many fixes to the Tests API. As mentioned above, this is currently being worked on by the dev team.

    Regards,
    Paul
     
  6. Phil R

    Phil R Established Member

    Thanks, I was going to suggest some other resource is embedded in test other than "user", but then I did some testing to get some of my answers. However, many of my concerns with "user" still apply.

    There are several reasons from a technical and practical elements to why you should not use "user".

    For one, I have now learnt about the "X_minimal" resource types, which are completely undocumented. Examples, such as "team_minimal", which I can only see the content through trial and error (the only ref in the docs, are in response examples).

    Technically

    1:
    The "user" resource is not very lightweight. It has many other embedded resources, which are highly detailed.

    In a test example, this could result in (from a quick inspection) resources embedded 4 deep (Test > User > Organization > Field > Field Option). In unit / functional testing, I would need to have implemented all these resources, just to prove a test.

    2:
    A "test" resource can be accessed from authenticated & unauthenticated sources, a function which should be retained.

    In an unauthenticated situation, a "user_minimal" resource is returned, again adding non-predictability.

    Practically

    1:
    Although it is a technical concern as well, the resource should return very predictable results.

    Right now, "test" will return a single test instance, or "count" instances.

    If you include "user", it become highly irregular, such as a "role" without nesting, "organization", "teams", "custom_fields" all with further irregular deep nesting.

    With predicable tests, I should be able to store a local copy (pasted within my unit test) of the nested json in the "data" block, parse that json along with the json returned by an actual call to the API and compare the results.

    The regulatory of user would prohibit that.

    2:
    Which user exactly do you intend to return?

    a) Would this be a specific user id every time, such as id "1"? How do you handle removed users?

    b) Would this be the user that made the authenticated request (given that basic, kayako and oauth all link to a specific account)?

    Again, what resource is returned in a situation in which no authentication is used?

    Legal

    Tests should exist in isolation.

    To your customers, their support desk is both productions, development and test platforms.

    In accordance with DPA requirements, plus best practice for mock development, tests should not be conducted on live data.

    I have the luxury that I am testing this on my trial instance, so all data is mock. However in production, I will be presented with live data under test conditions.

    Security


    Having noted the "X_minimal" resources, I have spotted some concerns with this.

    An an unauthenticated user, I am looking at "/articles". If I set "include=user_minimal,role,team_minimal", I now have access to detail associated with a staff member I probably should not.

    I can tell who as a "OWNER", "ADMIN", what teams exist.

    There is what i consider to be some quite revealing information within it.
     

Share This Page