Stack Exchange API: Stack Apps updates and upcoming version 3 beta

The new API is simpler and more conventional, which are nice properties to have. However the flexibility of the old API (and its "filter" system) was really powerful and made a lot of things easier and more efficient for users. I imagine it could sometimes have been irritating and complicated to support, but losing it would be a big step backwards. Here's a simple example of how I used it.

I sometimes like to save complete copies of a threads, with all of their answers and comments and such, to make sure I have a copy indexed and can find all of the information in the future. Typically to do this, I would grab the entire JSON API response for the question page, save it as POST_ID.json, then run it through an HTML template and save that alongside it as POST_ID.html, so I have it in a human-readable format but I still have the original full-fidelity data in case I want to update my template to include more details, or it had a bug or something.

For a specific example, consider Stack Overflow question 14952052: Convert blob URL to normal URL. With the current API, by defining the right filter I can get all of this information in a single request:

curl -o 14952052.json 'https://api.stackexchange.com/2.3/questions/14952052?order=desc&sort=activity&site=stackoverflow&filter=1x03bKxqqJ1t(qO_3rc0107HAS7gTDcx1bXQBo'

{
  "items": [
    {
      "tags": [...],
      "comments": [...],
      "answers": [
        {
          "comments": [
            {
              "owner": {
                "badge_counts": {...},
                "account_id": 420975,
                "reputation": 1290,
                "user_id": 798955,
                "user_type": "registered",
                "accept_rate": 90,
                "profile_image": "https://www.gravatar.com/avatar/0b145a77e0aaa86fbb591f5c639c42a4?s=256&d=identicon&r=PG",
                "display_name": "bhh1988",
                "link": "https://stackoverflow.com/users/798955/bhh1988"
              },
              "edited": false,
              "score": 27,
              "creation_date": 1409759592,
              "post_id": 16179887,
              "comment_id": 40077159,
              "content_license": "CC BY-SA 3.0",
              "body_markdown": "If blob urls don't point to server data, then how come Youtube videos' src url look like:\n\nsrc=\"blob:https%3A//www.youtube.com/44f26667-03f1-4978-9eed-af0cbf11dd67\"\n\n(in Chrome)",
              "body": "If blob urls don't point to server data, then how come Youtube videos' src url look like:  src="blob:https%3A//www.youtube.com/44f26667-03f1-4978-9eed-‌​af0cbf11dd67"  (in Chrome)"
            },
         ...

I haven't used the new API yet, but I'm trying to estimate what would be required based on the docs.

(These auto-generated docs, by the way, are not good: it defines a bunch of response types at the bottom, but the methods don't explicitly indicate which response types they return. I can infer it, but the existing docs are clearer about this. The only comment response type is "paginated comments", but there are no pagination parameters accepted by the question comments method, so is it paginated or not?)

I believe that instead of having a terminal one-line to grab the data, I'd need a script that would end up making requests for...

# the question itself:
- https://api.stackexchange.com/v3/stackoverflow/questions/14952052
# the comments on the question:
- https://api.stackexchange.com/v3/stackoverflow/questions/14952052/comments
# (possibly more requests if required by pagination, docs unclear)
# the answers on the question:
- https://api.stackexchange.com/v3/stackoverflow/questions/14952052/answers
# (potentially more requests if required by pagination)
# a request for the comments on every answer individually
- https://api.stackexchange.com/v3/stackoverflow/questions/14952052/answer/16179887/comments
- https://api.stackexchange.com/v3/stackoverflow/questions/14952052/answer/31581117/comments
- https://api.stackexchange.com/v3/stackoverflow/questions/14952052/answer/57141467/comments
- https://api.stackexchange.com/v3/stackoverflow/questions/14952052/answer/64857135/comments
- https://api.stackexchange.com/v3/stackoverflow/questions/14952052/answer/67080855/comments
- https://api.stackexchange.com/v3/stackoverflow/questions/14952052/answer/70918690/comments
- https://api.stackexchange.com/v3/stackoverflow/questions/14952052/answer/50303373/comments
# (and possibly more requests if required by pagination, docs unclear)

This is way more effort, way less efficient, and this still misses a ton of the metadata that's present in the old API.

Given the reality that the community has fled the site, it's obvious that the semi-engaged users remaining will not be updating all of the community tools and user scripts to use the new API (particularly when a replacing a single API request could require more code than some entire browser extensions already have). Community tooling is going to break and never recover.

Obviously they're not the target audience, and this way of defining an API is probably a bit easier for corporate users who can afford to spend more time and resources on integrations. For your business goals, this might be reasonable. But this is clearly another reflection of the company's lack of awareness or concern for the existing community and what has been built up over the site's lifetime. If you were just moving future development to the new API, that would be one thing, but if entirely dropping support for the new one, the only way folks will feel about that is upset.

(I don't think this was a primary motivation (the business case for this change is already clear enough), but I need to mention that this obviously makes it a lot harder for folks to scrape the site through the API, and that might could been a bonus for some folks making these decisions, without considering the impact on legitimate uses.)

Aside: I'm glad the new API doesn't HTML-escape by default. That was still a questionable choice when the API was new, but it was at least arguably, while today it seems very silly. Dropping accept_rate is also probably long-overdue — in this test I was surprised to see that it was still returning a meaningful value in the old API, instead of just a placeholder 0.

Licensing

The API does not appear to return any licensing information. I would expect that each Question, Answer, and Comment would be clearly and unambiguously marked with the license. I would expect it to be named ("CC BY-SA 4.0" or perhaps the SPDX identifier "CC-BY-SA-4.0") and use the Creative Commons Canonical URL (https://creativecommons.org/licenses/by-sa/4.0/) to ensure that recipients understand the terms of using the data.

Under the terms of CC BY-SA 4.0, attribution requires a notice that refers to the public license (Section 3(a)(1)(A)(iii)). Because this is not included, Section 6(a) is triggered and SE's rights terminate automatically. If not resolved in 30 days (as defined in Section 6(b)(1)), the right to use the material is terminated until express reinstatement (Section 6(b)(2)). Earlier versions of the license, such as 2.5 and 3.0, don't have the 30-day period to resolve issues (see also: the summary of changes in the 4.0 CC licenses).

To me, this is a huge showstopper for a beta. User-contributed content is going to be available to a broader audience without the appropriate licensing information, which is a violation of the CC licenses. I believe that this is crucial to resolve before expanding the beta.

Revisions

We're now getting closer to the beta, and there's some feedback from the moderator preview (staff/mod only link) that remains unresolved. Since it's still missing close to the beta, you're hearing about it.

The way you access revisions in this API needs some attention. Working with revisions is painful, whether it's through the data dump, SEDE, or this API. Since the data dump and SEDE are based on the database schema, I wouldn't necessarily expect them to change. But some of the problems can be massaged out in the API to make it more user-friendly.

The issue is that to get a post (especially a question, which has a title, a body, and tags) at a particular point in time, you need to walk the history of the post. In the definition of Question revision, if a revision only changed the body, lastBodyMarkdown and bodyMarkdown would be populated, but the lastTags, tags, lastTitle, and title fields would be null. So one would need to go to the previous revision, potentially all the way back to the original post, to get the state of a post at a particular point in time.

I see a few possible implementations:

1. Add a parameter to the revision endpoints (or perhaps the question and answer endpoints) that allows for specifying a date. If the post existed at that date, return all of the fields as they were at that particular date and time. If the post did not exist, return nulls.
2. Each Question revision or Answer revision should contain all of the fields. If a field was not modified, both lastX and X would have the same value. For example, if a revision modified the question title, the lastTitle would have the previous revision's title and title would have the revision's title. At the same time lastTags and tags would contain the same values, indicating the field hasn't changed.
3. Instead of sending the previous and current values, send the current revision's fields along with a boolean to indicate if it has changed since the previous revision. For example, if a revision modified the body, the revision object would contain a body that is the post-edit body and a boolean bodyModified of true. If someone wanted to do a comparison, they could then get the previous revision and compare the body.

Web URL, Share Link, and Attribution Link

What is the difference between the webUrl, shareLink and attributionLink? The examples for both questions and answers show an example that is the same for all three fields. Perhaps this is a documentation issue, but I would want to understand the differences in Swagger docs better.