How to Build Real-Time Voice Search on React

Author profile picture

@ottomatiasOttomatias Peura

Builds digital experiences.

Introduction

This tutorial will help you to get up and running with Speechly by guiding you through the process of building a simple voice filtering web app with Speechly and React.

You can find the source code for this tutorial on GitHub and you can also try out the final result.

Prerequisites

Since we’ll be using create-react-app for this tutorial, we’ll need the following tools:

– Node.js 8.10+
– npm 5.2+

Note that this tutorial also uses TypeScript, so feel free to check out TypeScript documentation if you’re not familiar with it.

1. Creating an app

Let’s get started by creating an app and installing its dependencies:

npx create-react-app speechly-voice-filter --typescript
cd speechly-voice-filter
npm i

Now that you’ve created the app, you can check it out by running `npm start` – it should open a browser tab with your app running in it.

2. Adding data and layout

Since we are building a filtering app, let’s add some data to filter and layout to display it.

To make it simple, our data source will be just a static array with some popular repositories on GitHub. Let’s add the following code and save it as `src/data.ts`:

ts
export type Repository = {
  name: string;
  description: string;
  language: string;
  followers: number;
  stars: number;
  forks: number;
};

export const repositories: Repository[] = [
  {
    name: "microsoft/typescript",
    description:
      "TypeScript is a superset of JavaScript that compiles to clean JavaScript output",
    language: "TypeScript",
    followers: 2200,
    stars: 65000,
    forks: 8700,
  },
  {
    name: "nestjs/nest",
    description:
      "A progressive Node.js framework for building efficient, scalable, and enterprise-grade server-side applications on top of TypeScript & JavaScript (ES6, ES7, ES8)",
    language: "TypeScript",
    followers: 648,
    stars: 30900,
    forks: 2800,
  },
  {
    name: "microsoft/vscode",
    description: "Visual Studio Code",
    language: "TypeScript",
    followers: 3000,
    stars: 105000,
    forks: 16700,
  },
  {
    name: "denoland/deno",
    description: "A secure JavaScript and TypeScript runtime",
    language: "TypeScript",
    followers: 1700,
    stars: 68000,
    forks: 3500,
  },
  {
    name: "kubernetes/kubernetes",
    description: "Production-Grade Container Scheduling and Management",
    language: "Go",
    followers: 3300,
    stars: 70700,
    forks: 25500,
  },
  {
    name: "moby/moby",
    description:
      "Moby Project - a collaborative project for the container ecosystem to assemble container-based systems",
    language: "Go",
    followers: 3200,
    stars: 58600,
    forks: 16900,
  },
  {
    name: "gohugoio/hugo",
    description: "The world’s fastest framework for building websites",
    language: "Go",
    followers: 1000,
    stars: 47200,
    forks: 5400,
  },
  {
    name: "grafana/grafana",
    description:
      "The tool for beautiful monitoring and metric analytics & dashboards for Graphite, InfluxDB & Prometheus & More",
    language: "Go",
    followers: 1300,
    stars: 37500,
    forks: 7600,
  },
  {
    name: "pytorch/pytorch",
    description:
      "Tensors and Dynamic neural networks in Python with strong GPU acceleration",
    language: "Python",
    followers: 1600,
    stars: 43000,
    forks: 11200,
  },
  {
    name: "tensorflow/tensorflow",
    description: "An Open Source Machine Learning Framework for Everyone",
    language: "Python",
    followers: 8300,
    stars: 149000,
    forks: 82900,
  },
  {
    name: "django/django",
    description: "The Web framework for perfectionists with deadlines",
    language: "Python",
    followers: 2300,
    stars: 52800,
    forks: 22800,
  },
  {
    name: "apache/airflow",
    description:
      "Apache Airflow - A platform to programmatically author, schedule, and monitor workflows",
    language: "Python",
    followers: 716,
    stars: 18500,
    forks: 7200,
  },
];
```

We can display this data in a simple table, so let's add a component for that under `src/RepoList.tsx`:

```tsx
import React from "react";

import { Repository } from "./data";

type Props = {
  repos: Repository[];
};

export const RepoList = ({ repos }: Props): JSX.Element => {
  return (
    <div className="block">
      <table>
        <thead>
          <tr>
            <th>Name</th>
            <th>Language</th>
            <th>Description</th>
            <th>Followers</th>
            <th>Stars</th>
            <th>Forks</th>
          </tr>
        </thead>
        <tbody>
          {repos.map((repo) => (
            <RepoRow repo={repo} key={repo.name} />
          ))}
        </tbody>
      </table>
    </div>
  );
};

const RepoRow = React.memo(
  ({ repo }: { repo: Repository }): JSX.Element => {
    return (
      <tr>
        <td>{repo.name}</td>
        <td>{repo.language}</td>
        <td>{repo.description}</td>
        <td>{repo.followers}</td>
        <td>{repo.stars}</td>
        <td>{repo.forks}</td>
      </tr>
    );
  }
);

In order to show the table, we’ll need to render it. We could render our table right in our top-level `App` component, but let’s instead use a top-level component for our app under `src/SpeechApp.tsx`, it will come in handy later on:

import React from "react";

import { repositories } from "./data";

import { RepoList } from "./RepoList";

export const SpeechApp: React.FC = (): JSX.Element => {
  return (
    <div>
      <RepoList repos={repositories} />
    </div>
  );
};
```

Now let's add it to our top-level component:

```tsx
import React from "react";
import { SpeechProvider } from "@speechly/react-client";

import "./App.css";

import { SpeechApp } from "./SpeechApp";

function App(): JSX.Element {
  return (
    <div className="App">
      <SpeechApp />
    </div>
  );
}

export default App;

3. Adding Speechly client and a microphone button

Before we proceed with the app, let’s take a quick detour and train a very simple and not very useful Speechly app, so that we can use it to test our integration later on.

Go to https://www.speechly.com/dashboard and login (or sign up if you haven’t yet) and create a new app (you can check our Speechly Dashboard quickstart guide if you feel lost). Feel free to use any configuration you want, even an almost empty configuration with just a “Hello world” will suffice, but make sure your app is deployed!

Once you have your Speechly app deployed, let’s integrate it. Start by installing Speechly React client:

“`sh

npm i –save @speechly/react-client

“`

The client exposes a context provider and a hook that allows you to consume that context. Let’s add the context provider to `src/App.tsx` – make sure you provide the `App ID` of your Speechly app as a property for `SpeechProvider`!

“`tsx

import React from “react”;

import { SpeechProvider } from “@speechly/react-client”;

import “./App.css”;

function App(): JSX.Element {

return (

<div className=”App”>

<SpeechProvider appId=”your-app-id-here” language=”en-US”>

<SpeechApp />

</SpeechProvider>

</div>

);

}

export default App;

“`

Next let’s add some code to act as the microphone button. Also, it would be nice to see what we are saying, so let’s also render the transcript next to the button for some feedback. Let’s make that a separate component and save it as `src/Microphone.tsx`:

import React from "react";
import {
  Word as SpeechWord,
  SpeechSegment,
  SpeechState,
} from "@speechly/react-client";

type Props = {
  segment?: SpeechSegment;
  state: SpeechState;
  onRecord: () => Promise<void>;
};

export const Microphone = React.memo(
  ({ state, segment, onRecord }: Props): JSX.Element => {
    let enabled = false;
    let text = "Error";

    switch (state) {
      case SpeechState.Idle:
      case SpeechState.Ready:
        enabled = true;
        text = "Start";
        break;
      case SpeechState.Recording:
        enabled = true;
        text = "Stop";
        break;
      case SpeechState.Connecting:
      case SpeechState.Loading:
        enabled = false;
        text = "Loading...";
        break;
    }

    return (
      <div>
        <button onClick={onRecord} disabled={!enabled}>
          {text}
        </button>
        <Transcript segment={segment} />
      </div>
    );
  }
);

const Transcript = React.memo(
  ({ segment }: { segment?: SpeechSegment }): JSX.Element => {
    if (segment === undefined) {
      return (
        <div>
          <em>Waiting for speech input...</em>
        </div>
      );
    }

    return (
      <div>
        {segment.words.map((w) => (
          <Word word={w} key={w.index} />
        ))}
      </div>
    );
  }
);

const Word = React.memo(
  ({ word }: { word: SpeechWord }): JSX.Element => {
    if (word.isFinal) {
      return <strong>{`${word.value} `}</strong>;
    }

    return <span>{`${word.value} `}</span>;
  }
);

As you can see this component renders a button that calls the `onRecord` callback passed in the properties and uses the state of Speechly client to determine when to enable the button and which text to use as its label. In addition to that the component also renders the transcript of the phrase by assembling individual transcripted words from a segment (check out [this article in our documentation](/speechly-api/#understanding-server-responses) for more information about how SLU API works). Since a word can be either tentative (i.e. its value can change as the API receives more audio data) or final, we use bold text to highlight final words.

One more step – we’d need to render our component and hook it up to the API. Let’s add it to our `SpeechApp` component:

import React from "react";
import { useSpeechContext } from "@speechly/react-client";

import { repositories } from "./data";

import { RepoList } from "./RepoList";
import { Microphone } from "./Microphone";

export const SpeechApp: React.FC = (): JSX.Element => {
  const { toggleRecording, speechState, segment } = useSpeechContext();

  return (
    <div>
      <Microphone
        segment={segment}
        state={speechState}
        onRecord={toggleRecording}
      />
      <RepoList repos={repositories} />
    </div>
  );
};

Here we use the other main part of Speechly React client – a custom hook that consumes the state preserved in `SpeechProvider`. Feel free to check the [API documentation of React client](https://github.com/speechly/react-client/blob/master/docs/modules/_index_d_.md) to see what other properties are returned by the hook.

Now you can go ahead and try talking to the app and see what you get back in the transcript. Congratulations, you’ve just integrated Speechly into the app.

However, we still need to implement the filtering functionality, so let’s go ahead and update our Speechly app configuration to support that.

4. Configuring Speechly app

Now that we’ve integrated the API into the app it’s time to make our Speechly app useful. Let’s add a couple of simple commands for manipulating the data we see in the table:

– A command to filter by programming language, e.g. when a user says “Show me TypeScript repos” the app will only show repos with that specific language

– A command to sort the results in a specific order, e.g. “Sort the results by forks” will sort the repos by the amount of forks it has.

– A command to reset the filters, e.g. “Reset the filters to default” will remove the language filter and reset the sorting to some default.

Let’s go back to Speechly dashboard and update the configuration of our app with the following:

# Which languages we can filter by
languages = [
  Go
  TypeScript
  Python
]

# Which fields we can sort by
sort_fields = [
  name
  description
  language
  followers
  stars
  forks
]

# Synonyms for "repo"
results = [
  items
  results
  repos
  repositories
]

# A couple of commands for filtering.
#
# This will expand into e.g. following examples (not exhaustive):
# "Show all Go repos"
# "Show me only TypeScript repositories"
# "Show Python results"
# etc.
#
# Words in curly brackets ("{me}") are optional.
# Square brackets are for lists (e.g. one option from the list may be used)
*filter show {me} {[all | only]} $languages(language) {$results}
*filter filter {$results} by $languages(language) {language}

# A command for sorting, e.g.:
# "Sort the repos by name"
# "Order results by forks"
# etc.
*sort [sort | order] {the} {$results} by $sort_fields(sort_field)

# A command for resetting the filters, e.g.:
# "Reset all filters to default"
# "Remove the filters"
# "Reset to default"
# etc.
*reset [reset | remove] {[the | all]} {filters} {to default}

Don’t forget to add `sort`, `filter` and `reset` as intents and `languages` and `sort_fields` as entities!

As you can see from the comments, this configuration will make our Speechly app understand the commands we need and properly detect entities and intents. Keep in mind, that the cool part is that the model will also be able to understand the variations of commands that are not explicitly defined in our configuration. The same also applies to entities – the app won’t be limited to only detecting “Go”, “TypeScript” and “Python” as options for the language, but other words as well, which will be roughly in the same place in a pharse (e.g. you could try saying “Show me all Javascript repos”). However, with words that are very specific to domain like programming language names it’s always a good idea to list them all in your configuration, otherwise they might be mistaken for some regular words, e.g. the API might not properly detect “Rust” as a programming language if you say “Show me all Rust repositories”, because it would think that you meant “rust” as that thing that destroys metals. You can read more about how to configure Speechly applications.

Once you’ve deployed your new version of the Speechly app, let’s continue to parsing the results.

5. Parsing intents and entities

Now that we’ve trained a version of Speechly app with proper entities and intents, let’s parse the results. First let’s add our parsing logic to `src/parser.ts`:

ts
import { SpeechSegment } from "@speechly/react-client";

export enum IntentType {
  Unknown = "unknown",
  Sort = "sort",
  Filter = "filter",
  Reset = "reset",
}

export enum EntityType {
  Language = "language",
  SortField = "sort_field",
}

export enum SortEntityType {
  Unknown = "unknown",
  Name = "name",
  Description = "description",
  Language = "language",
  Followers = "followers",
  Stars = "stars",
  Forks = "forks",
}

const SpeechIntentValues = Object.values(IntentType) as string[];
const SortTypeValues = Object.values(SortEntityType) as string[];

export function parseIntent(segment: SpeechSegment): IntentType {
  const { intent } = segment;

  if (SpeechIntentValues.includes(intent.intent)) {
    return intent.intent as IntentType;
  }

  return IntentType.Unknown;
}

export function parseLanguageEntity(segment: SpeechSegment): string[] {
  const langs: string[] = [];

  for (const e of segment.entities) {
    if (e.type === EntityType.Language) {
      langs.push(e.value.toLowerCase());
    }
  }

  return langs;
}

export function parseSortEntity(segment: SpeechSegment): SortEntityType {
  let s = SortEntityType.Unknown;

  for (const e of segment.entities) {
    const val = e.value.toLowerCase();

    if (e.type === EntityType.SortField && SortTypeValues.includes(val)) {
      s = val as SortEntityType;
    }
  }

  return s;
}

Here we define a couple of functions to parse intents and different entity types from a `SpeechSegment`, which is returned by `useSpeechContext`. As you can see, the code is pretty straightforward, most of it is actually just listing which intents and entities we expect and defining them as enumerations, since it’s always a good idea to check the results returned from API against a pre-defined list of allowed values to avoid bugs. Another good idea is to make sure we use consistent case (in this case by casting the results to lower case) to avoid false negatives when e.g. comparing `STARS` to `stars`.

Now that we have our code for parsing the results from a segment, time to use it. Let’s update our `SpeechApp` and add some code that calls our parser:

import React, { useEffect } from "react";
import { SpeechSegment, useSpeechContext } from "@speechly/react-client";

import { repositories } from "./data";
import {
  IntentType,
  SortEntityType,
  parseIntent,
  parseLanguageEntity,
  parseSortEntity,
} from "./parser";

import { RepoList } from "./RepoList";
import { Microphone } from "./Microphone";

export const SpeechApp: React.FC = (): JSX.Element => {
  const { toggleRecording, speechState, segment } = useSpeechContext();

  useEffect(() => {
    if (segment === undefined) {
      return;
    }

    parseSegment(segment);
  }, [segment]);

  return (
    <div>
      <Microphone
        segment={segment}
        state={speechState}
        onRecord={toggleRecording}
      />
      <RepoList repos={repositories} />
    </div>
  );
};

function parseSegment(segment: SpeechSegment) {
  const intent = parseIntent(segment);

  switch (intent) {
    case IntentType.Filter:
      const languages = parseLanguageEntity(segment);
      console.log("Filtering by languages", languages);
      break;
    case IntentType.Sort:
      const sortBy = parseSortEntity(segment);
      if (sortBy !== SortEntityType.Unknown) {
        console.log("Sorting by field", sortBy);
      }
      break;
    case IntentType.Reset:
      console.log("Resetting the filters");
      break;
  }
}

Here we define a `parseSegment` function that is called every time a segment changes by using React’s `useEffect` hook. Since segment might come as `undefined` (this happens after the the user stops speaking and the API sends it’s final response), we want to check for that before trying to parse it. The function checks for the intent and then calls the appropriate entity parser (or no entity parser at all if the intent was to reset the filters). For now we are just going to log the results of the parser, but to use them we’ll have to add some filters. Let’s continue with that!

6. Adding and applying filters

In order to apply filters, we’d need to implement some filtering logic, so let’s do just that and add it as `src/filter.ts`:

Here we define a `Filter` type that contains a list of languages to display and the field to sort by. We also define a function `filterRepos` that takes a list of repositories and a filter and returns a new list of repositories filtered and sorted according to that filter.

Now we need to call the filtering function when we get new results from the API, so let’s also update our `SpeechApp` to do that:

tsx
import React, { useEffect, useState } from "react";
import { SpeechSegment, useSpeechContext } from "@speechly/react-client";

import { repositories, Repository } from "./data";
import { Filter, filterRepos } from "./filter";
import {
  IntentType,
  SortEntityType,
  parseIntent,
  parseLanguageEntity,
  parseSortEntity,
} from "./parser";

import { RepoList } from "./RepoList";
import { Microphone } from "./Microphone";

export const SpeechApp: React.FC = (): JSX.Element => {
  const [filter, setFilter] = useState<Filter>(defaultFilter);
  const [repos, setRepos] = useState<Repository[]>(repositories);

  const { toggleRecording, speechState, segment } = useSpeechContext();

  useEffect(() => {
    if (segment === undefined) {
      return;
    }

    const nextFilter = {
      ...filter,
      ...parseSegment(segment),
    };

    setFilter(nextFilter);
    setRepos(filterRepos(repositories, nextFilter));
    // eslint-disable-next-line react-hooks/exhaustive-deps
  }, [segment]);

  return (
    <div>
      <Microphone
        segment={segment}
        state={speechState}
        onRecord={toggleRecording}
      />
      <RepoList repos={repos} />
    </div>
  );
};

const emptyFilter: Filter = {};
const defaultFilter: Filter = {
  languages: [],
  sortBy: SortEntityType.Name,
};

function parseSegment(segment: SpeechSegment): Filter {
  const intent = parseIntent(segment);

  switch (intent) {
    case IntentType.Filter:
      const languages = parseLanguageEntity(segment);

      if (languages.length === 0) {
        return emptyFilter;
      }

      return {
        languages,
      };
    case IntentType.Sort:
      const sortBy = parseSortEntity(segment);
      if (sortBy !== SortEntityType.Unknown) {
        return {
          sortBy,
        };
      }

      return emptyFilter;
    case IntentType.Reset:
      return defaultFilter;
    default:
      return emptyFilter;
  }
}

Here we use React’s `useState` hook to create a couple of stateful variables for storing filtered results and last filters (since you can append them by saying “Show me all Go repos” first and then following up with “Sort by start”). Every time we get new state of `segment` from the API, we call our `parseSegment` to parse the filters from it and then append those filters to the ones we’ve saved in the state. Then we also apply new filters to the list of repositories before passing them on to rendering.

Conclusion

And that’s it! Now you can go ahead and try out your app – you can filter the repos by language, apply some sorting order and reset the filters.

If you want to delve into the details, go ahead and check out our documentation and our public https://github.com/speechly

You can also check the source code for this tutorial at https://github.com/speechly/react-example-repo-filtering. Feel free to navigate through individual commits – they refer to each section of this tutorial.

Also published on Speechly.

Tags

The Noonification banner

Subscribe to get your daily round-up of top tech stories!

Next stable version of GNOME will be called GNOME 40

Interview Patent assertion entities: do not pick a fight with open source. It won’t end well for you. This is the message from GNOME Foundation executive director Neil McGovern, who will speak on the subject at the Open Source Summit Europe next week.

McGovern talked to The Register ahead of the event on patents, Microsoft, and more.

The open-source outfit develops the default desktop environment on major Linux distributions including Ubuntu and Red Hat. In late August 2019, Rothschild Patent Imaging filed a lawsuit against the GNOME foundation claiming that GNOME Shotwell, a photo manager, infringed one of its patents.

“We didn’t receive a letter before the court documents were filed or any sort of warning, it was just filed and then within a week there was a settlement request for $75,000,” McGovern told us.

“I guess it would have been easier if we just settled and moved on, but I felt that wasn’t the right thing to do, and would have been a big problem in the future.”

GNOME executive director Neil McGovern

Instead, GNOME Foundation said it would “vigorously defend against this baseless suit.” Nine months later, the dispute was settled without payment, and “both Rothschild Patent Imaging and Leigh Rothschild are granting a release and covenant to any software that is released under an existing Open Source Initiative approved license.”

McGovern said he was “exceptionally pleased with the outcome… I felt it was incredibly important to send a message to the entire patent assertion industry that basically you don’t go after open source projects. It won’t end well for you.”

What enabled this achievement? Much of it comes down to funding, said McGovern. “Normally the way these entities work is that they want a quick payout and to move on. They don’t want it to go to court.”

The idea of running a full free software stack on your phone, and people being in control of their computing on their phone, is something that has been lost, certainly for the moment

The problem though is that “to fully defend a pattern infringement case it usually costs about $10,000,000,” he said.

GNOME though got pro-bono representation from Shearman & Sterling LLP which was “a huge help,” said McGovern.

The Foundation also raised public funds, to meet “other costs that we had in defending it, flights, expert witnesses, patent searches, court filing fees, etc,” he said.

Microsoft, Windows Subsystem for Linux, and mobile devices

Moving onto matters new, Microsoft is working with Canonical to get a Linux GUI, no doubt GNOME based, running on Windows 10 via Windows Subsystem for Linux (WSL). What is the significance of that?

“I’m certainly very pleased to see that Microsoft are moving more towards encouraging open source development,“ said McGovern. “With regards to WSL in particular, I don’t see it as a threat. Our priorities are around encouraging end user control over their own computer, end user freedom.

“I think the only way that that can happen is through use of a truly open source stack. A proper open source desktop.”

There is an issue in that despite rising use of Linux and open source, end user computing environments are getting more locked down. “In some ways, open source has won, it is ubiquitous, I think now 100 of the top 100 supercomputers are now running Linux,” said McGovern. “But there’s still that challenge in that this isn’t the sort of Utopia we all wished for when we first started doing this, people being able to control their own computing hasn’t yet been reached.”

One of GNOME Foundation’s key challenges is “trying to ensure that Linux on the desktop grows and remains relevant in a world where we’re seeing more and more services go online, where use of mobile computing [and] cloud computing is growing,” said McGovern.

He observes though that “the use of desktop computing, and I include laptops here, has actually increased, but there has been a huge rise in mobile devices. But it’s the thing people use when they want to get real work done, like writing an essay or doing your accounts.”

Is the big problem though that GNOME has lost on mobile, just as Microsoft has done? “We have things like Purism producing the Librem mobile phone,” said McGovern.

“But the idea of running a full free software stack on your phone, and people being in control of their computing on their phone, is something that has been lost, certainly for the moment. There is dominance from iOS and Android. It has become very expensive and complex to produce hardware and then ship something on it, as Nokia and Blackberry and Microsoft have found out.

“It’s hugely problematic. We don’t have the funding to try and drag that forward. If we had a big pot of money I’d love for us to be able to produce a mobile ourselves, but that’s not where we are at the moment.

Snaps and Flatpaks

Flathub, GNOME’s community-driven answer to Canonical’s Snap Store

Flathub, GNOME’s community-driven answer to Canonical’s Snap Store

The developers of Linux Mint are troubled about Canonical’s Snap Store, what does McGovern think? “I don’t have an issue with Snaps,” he said. “I do have a concern that the Snap Store is entirely gated by Canonical.”

“The effort that is going in to have application developers be able to target a single runtime or that works across distributions, I think is absolutely fantastic, and it’s something that’s been missing for many many years on Linux systems. For an app developer to go ‘Hey, I want to make it available for MacOS and then for Windows and then these 30 different Linux distributions which all work in a special way’ is just not something that is going to drive that ecosystem towards Linux.”

The GNOME answer is Flatpaks and an associated repository called Flathub. “Unlike with Snaps, people can set up their own Flatpak repositories, so it becomes a lot more distributed in that people set up whatever they want. With Flathub, we’re truly trying to make it a cross platform thing, so it’s not just us, KDE have their stuff on there. And also including some proprietary applications, make a place where there is a App Store, but it is run on behalf of the community,” he said.

GNOME versioning: no repeat of 2 to 3 disruption

The current version of GNOME is 3.38, but the next, expected around March 2021, will be called GNOME 40. You could think of it as 3.40 because the Foundation is determined to avoid a repeat of the disruption caused when GNOME 3.0 appeared in 2011. The future will be regular small changes, not big bang releases, as explained by the Foundation’s Emmanuele Bassi.

“We’ve reached the point where we have a platform that we’re reasonably happy with,” said McGovern. “There isn’t really a need for a big overhaul in the same way that that happened before. GNOME 2 and its technology stack was very limiting in what could be done, and there was a lot of old code there. Things are moving towards a more iterative approach.”

There will be no GNOME 4.0 because, “if we ever did release 4, then people would see it as a huge change in [that] everything’s going to be broken again, and that’s not really what we’ve got,” he said.

Even changes like the move from X to Wayland for the display protocol, which is a big technical change, can be done without disruption to users. “The idea is that users should not necessarily notice a difference, except that things will be smoother and potentially more fluid because they can actually use proper 3D acceleration,” said McGovern. “Our aim is to ensure that stuff which worked before continues to work. It’s actually a lot more secure way for people to use their computer. X had a lot of issues with its architecture.”

GNOME not just a desktop

The GNOME 3 desktop

The GNOME 3 desktop

GNOME, said McGovern, is not just a desktop. “Everything from libxml which is used everywhere, D-Bus and things like GStreamer which is the audio visual framework that’s used in millions of TV’s and cars, is something that’s come from GNOME itself because to produce this simple, easy to use desktop environment we have had to have lots of plumbing underneath it.

“As a foundation as well, we are also trying to drive not just the desktop, we do Flathub, we’re currently running a Community Engagement Challenge which is about teaching people how to contribute to open source software. We need to be able to do this to secure that future of people’s control of their own computing, rather than just using a desktop. A lot of the stuff we do has a wider context.”

Defeating patent trolls is a great example. ®

Mark Kelly May Become First Astronaut to Make It to Congress on His First Try

Mark Kelly isn’t the first former NASA astronaut to run for office, but if he’s elected he’ll be the only one to make it to Congress on his first shot.

Call him the cosmic candidate. At 56 years old, former NASA astronaut Mark Kelly has logged nearly two months in space as the pilot or commander of four space shuttle missions. After retiring from the astronaut corps in 2011, Kelly helped make history as the control subject in an unprecedented study on the way space affects the human body, in which his twin brother Scott spent a record-breaking 340 consecutive days in orbit. And now, the astronaut and Navy veteran is making waves after emerging as the frontrunner in a high-stakes race for one of Arizona’s seats in the US Senate.

The Arizona race is a special election to fill the vacancy left by John McCain, who died of brain cancer in 2018. McCain’s seat was filled by gubernatorial appointment until this November, when Arizonans will decide who will finish the final two years of his term. Kelly, a Democrat, announced his candidacy in early 2019, and sought to position himself as the independent voice of reason. “I’m running for the United States Senate because Washington is broken,” Kelly wrote on his website. “Partisanship keeps politicians from finding solutions, and all of the money in our political system keeps politicians from being accountable to the people they’re supposed to represent.”

(His campaign did not respond to WIRED’s requests for comment.)

If Kelly wins, he would be among a small constellation of astronauts-turned-politicians, a coterie that includes two Apollo crew members and one space shuttle commander. On the trail, Kelly has played up his astronaut cred and scientific sensibilities. His campaign sells t-shirts that read “Science + Data + Facts,” bumper stickers featuring a space shuttle, and star-studded campaign buttons. It’s not hard to see why. Astronauts have been regarded as national heroes even before Buzz and Neil set foot on the moon. They’re the types of people kids want to be when they grow up. They’re daring adventurers who are willing to stare death in the face to push the boundaries of human knowledge and promote peace among nations. In short, astronauts are the exact opposite of whatever comes to mind when most people think of a politician.

“I think that most people are looking for independent representatives,” says Shaughnessy Naughton, the president and founder of 314 Action, a nonprofit organization that supports scientists running for office in the US and whose Action Fund has contributed to Kelly’s campaign. “They want honesty, they want transparency, and they want leaders that will base their conclusions on facts and evidence, and we’ve certainly seen that from Mark Kelly.”

In the early days of NASA’s human space program, some astronauts saw themselves as literally and figuratively above the trifling concerns of terrestrial politics. The sentiment was succinctly captured by Apollo 14 astronaut Edgar Mitchell after his journey to the lunar surface. “From out there on the moon, international politics look so petty,” Mitchell said in an interview with People magazine after he returned. “You want to grab a politician by the scruff of the neck and drag him a quarter of a million miles out and say, ‘Look at that, you son of a bitch.’”

But others realized that their space age sheen could also launch them into some of the most powerful positions in government. Just two years after he became the first American to orbit Earth as part of NASA’s Mercury program, John Glenn ran for the US Senate. “By the time that he made his flight in Mercury, he was a household name, and he played that for all it was worth,” says Roger Launius, the former chief historian at NASA.

Despite his historic achievement, Glenn’s initial foray into politics was harshly criticized from both sides of the aisle. “The high office of the US Senator from the State of Ohio should not be made a hero’s pawn, no matter the breadth of our gratitude,” said Representative Charles Vanik in a 1964 article in the St. Petersburg Times. “This grave responsibility, so vital to the state and to the nation, should not be vested to the unprepared.” The same article also mentioned “uneasy speculation” among other members of Congress that Glenn might kick off a trend of senators from space.

But even though Glenn hadn’t spent time in office, he did have a lot of skills that are useful for statecraft. “He was a master at negotiating the bureaucracies of the Marine Corps and NASA,” says Launius. “He also knew how to play to the public, which is important if you’re going to run for any sort of office. So from that standpoint, he was really good at this.”

Glenn’s political dreams were put on pause just months before the election in 1964 when he slipped and fell in a tub. He lost during his next campaign in 1970. But in 1974, Glenn won a special election and was elected as a Democratic Senator for Ohio, a position he held for the next 25 years. Aside from a failed campaign for president, Glenn’s political career was rather unremarkable. Most of the enacted bills he sponsored were for the creation of obscure national holidays, and Launius says he rarely ever broke ranks with his fellow Democrats. “He was a pretty reliable vote,” he says. “If the party was behind it, he was usually behind it.” Arguably, his most enduring legacy is setting a precedent for the astronaut-turned-politician.

Since Glenn’s election, a number of astronauts have run for Congressional office. Harrison Schmitt, a member of the last Apollo mission to the moon, served as a Republican Senator for New Mexico for a few years, and Jack Swigert, a member of the ill-fated Apollo 13 mission, won a seat in the House of Representatives after losing a bid for the US Senate, but he died before taking office. Being an astronaut isn’t a guaranteed ticket to Capitol Hill, however. Jack Lousma, a member of the second crew to visit Skylab, America’s first space station, lost a 1984 election to represent Michigan in the Senate. And Jose Hernandez, the most recent astronaut to run for Congress, lost to an almond farmer in California in 2012.

If Kelly is elected this November, he’ll be the only astronaut to make it to Congress on their first try. Throughout most of his campaign, Kelly has had a lead over Republican incumbent Martha McSally in the polls reported by the political analysis site FiveThirtyEight, although polls added on Friday show them in a dead heat. That Kelly maintained a lead for so long in a state that has gone Republican in all but one of the past 17 presidential elections is surprising, especially given his lack of experience in an elected office.

Some of that might have to do with an electorate that’s hungry for evidence-based policy during a presidential administration that wages war on science. Arizonans have especially borne the brunt of anti-scientific politics during the coronavirus pandemic. Doug Ducey, the state’s governor, was slow to shut down businesses and quick to reopen them in the early stages of the pandemic. Throughout the pandemic, Arizona has had an overall testing positivity rate of slightly over 14 percent, one of the highest in the nation.

“Communities are suffering and the main reason they’re suffering is we have a failure of leadership to address a serious pandemic,” Kelly said during a debate with McSally earlier this month. “I worked at NASA for 15 years, and NASA wouldn’t give you the 17th century solution to a 21st century problem.”

Still, Kelly’s campaign seemed like a longshot bid in a historically red state. He leans left on some policies—he advocates for healthcare as a right, ending tax breaks for large corporations, lowering interest on federal student loans, and more renewable energy—but is more centrist on others. As a gun owner and spouse of Gabrielle Giffords, a former US representative who was shot in the face during an assassination attempt, Kelly advocates for universal background checks, but also supports the right to bear arms. He says he wants strong borders, but also protection for DREAMers, children born in the US to parents who lack documentation. Kelly is not an archetypal Democrat—and he doesn’t want to be. He’s running on the idea that he’s a pilot, an engineer, and an astronaut, not a beltway bureaucrat.

“I’m someone who cares about independence, science, data and facts,” Kelly said during an appearance on The View last year. “I don’t always see that from Washington, DC, and we have some serious issues we’re facing and they’re not often being addressed.”

But his campaign’s focus on the virtues of being a Washington outsider is hardly unique. Most notably in recent politics, during Donald Trump’s 2016 bid for president he positioned himself as someone who could “drain the swamp,” overcoming the perceived impotence and ulterior motives of career politicians. Nichole Bauer, a political scientist at Louisiana State University and expert on the way political campaigns communicate, says this can be an effective strategy, but she thinks it’s one of the least compelling aspects of Kelly’s messaging.

“When voters are looking for a candidate, they’re really looking for somebody that can project power, strength, and authority,” says Bauer. “The best way to get those credentials is to have some sort of background in a stereotypically masculine profession. Kelly was in the military and he was an astronaut. There’s no other better way to project those things than what Kelly does.”

Kelly and his opponent have a lot in common. Both have had illustrious careers in the military and both have played up their outsider status in Washington. (McSally was a colonel in the Air Force and was the first US female squadron commander.) But now that she has served in the Senate for a year, it’d be hard for her to make the case she’s a true outsider. And even though both served in the military, Kelly has been to space. “The astronaut status gives him a leg up over McSally,” says Bauer. “If he was not an astronaut, he would definitely not have as much of a shot, because he wouldn’t have that celebrity background and cachet.”

Even in an election cycle that has seen an unusually large amount of money flowing into Senate races, the spending on the Arizona race still stands out. Kelly’s campaign has raised more than $83 million, an astronomical sum that’s more than four times higher than the average cost of winning a Senate seat, and McSally’s campaign has raised nearly $50 million. Senate campaign funds from both parties have each allocated well over $10 million on ad campaigns attacking the other candidate and out-of-state funding for both candidates tops the charts.

But this isn’t your average Senate race. Because it’s a special election, whoever Arizonans elect in November could take office before the end of the year, instead of waiting until January to be sworn in. That means Arizona’s new senator could provide a pivotal vote in the controversial Supreme Court nomination process of Amy Coney Barrett. While McSally has voiced her support for the nominee, during their debate this month Kelly said he’d vote no on Barrett’s nomination.

Arizona is a historically red state, but demographic trends have turned it purple over the past few years and some analysts think this might be the year it goes blue. To science advocates, the fact that voters in a traditionally conservative state would embrace a candidate like Kelly is a reminder that change is possible, even at a time when science is under attack at the highest levels of government. “Very few people have gone to space, and hearing about that experience is super exciting and inspiring,” says Naughton. “I think it also gives voters an extra level of trust and confidence that he does have the right motivation for taking on this challenge. It’s not just the next step in his congressional career. It’s that he sees a problem that he is well positioned to address and fix.”


More Great WIRED Stories

Show HN: Collection of tiny C ELF programs with graphics output

This repository is a collection of small ELF (Executable and Linkable Format) executables able to output graphics. All compiled with GCC 7.5.0.

The goal was to build a collection of very small standalone Linux graphics programs written in C with minimal amount of assembly (which is inline and mainly used for syscall)

The rules was to be able to output & view graphics data, it also should be able to quit properly using Ctrl+C (SIGINT)

All of them compile down to less than 512 bytes with some less than 256 bytes (182 bytes to be exact excluding ASM version)

There is only one pure assembly framebuffer program to show how all of this compete against pure assembly and compliant ELF (maybe one could reach < 128b by dropping compliance) credits

It borrow several tricks from several sources mainly coming from the Demoscene

All C programs work in either 64 bits or 32 bits (must append -m32 to GCC flags), 32 bits programs may be bigger or smaller.

This was used for my Twigs 512 bytes procedural graphics intro.

Build

Just go into any directory then into src directory and type sh build.sh this will invoke make multiple times for all defined width / height parameters in build.sh then all generated executables will go into the upper directory.

This was built with GCC 7.5.0

How

  • using GCC compiler optimizations through compiler flags (see Makefile of each programs)
  • -nostartfiles -nodefaultlibs linker options among others (don’t link to libc / don’t use default startup code)
  • usage of strip to remove various useless stuff from the executable binary (symbols, debug data etc.)
  • usage of sstrip to remove even more useless stuff that strip doesn’t (section headers)
  • clean more useless bits from the executable with a sed pass from blackle
  • compress the ELF binary with LZMA (Note: this step + next step of course rely on some tools like lzcat sed which must be available on the OS; this is generally not an issue as they are available by defaults on most Linux OS)
  • build a binary from a small shell script + the compressed binary, the shell script unpack and run itself, it unpack the executable to /tmp/g, make it executable and may perform additional stuff (such as outputting audio, clearing the terminal, exiting properly etc.)
  • truncate some bytes left from the compression format (CRC32 checksum)

Some details / credits about the optimizations can be gathered here

If the compression / shell script may feel like cheating one can still compile some programs (framebuffer, file) down to less than 256 bytes for file / fbdev output.

There is still some room to remove some bytes if one don’t care about clearing the terminal output or exiting properly. (~11 bytes saved)

There is also the -march= GCC option which can have varying result with ~2 bytes gain.

strace is usefull on optimized binary to see any problems with the syscall

Graphics output

Several methods are used to output graphics, of which :

GPU-only method using a GLSL fragment shader :

Framebuffer / file output is probably the most compatible option followed by SDL as i believe it is installed by default on many systems.

Sound output

There is no sound output on the provided samples but it can be added easily by using aplay in the shell script (and piping data to it such as obviously /dev/random)

File output

Only limited to static graphics data.

This output a .ppm (Portable Pixmap) image named i with a centered white pixel in current directory and call xdg-open to open it.

There is two versions of the program in file.c :

  • standard version with open / write syscall, it is slightly larger than the one below
  • (default) shortcut version which write the binary data to stdout (single write syscall) which is then redirected by the shell script to the .ppm file

64 bits ELF result :

  • 190 bytes optimized + compressed

Note :

  • Due to PPM format some more bytes may be taken when the image resolution is increased.
  • Some more bytes could be gained by calling eog or feh instead of xdg-open but it would probably reduce compatibility.
  • The image should be named .ppm to be compatible with most files explorer.

Framebuffer

This use the framebuffer device /dev/fb0 (fbdev) to output graphics data. (white centered pixel)

The framebuffer device sometimes require that the user must either be in video or tty group otherwise the executable must be run by root or with sudo

The generated binary res / bit depth should match the framebuffer settings in order to work.`

64 bits ELF result :

  • 240 bytes optimized
  • 182 bytes optimized + compressed

SDL

This use the SDL library to output graphics data. (white centered pixel)

Only two calls : SDL_SetVideoMode and SDL_Flip, SDL_Init call seem uneeded so it was left out.

Exiting properly was a bit tough through SDL calls so it was handled in the shell script by running the program in background, setting up a SIGINT trap which then kill the process.

64 bits ELF result :

  • 969 bytes optimized
  • 404 bytes optimized + compressed

Switching to 32 bits ELF actually free some more bytes :

32 bits ELF result :

SDL2

This use the SDL2 library to output graphics data. (white centered pixel)

Unless the SDL version this one doesn’t seem to require any additional setup to exit properly ?

64 bits ELF result :

  • 1073 bytes optimized
  • 456 bytes optimized + compressed

Clutter (accelerated)

This use the Clutter libary to output graphics data through a GLSL fragment shader. (white screen)

64 bits ELF result :

  • 1409 bytes optimized
  • 464 bytes optimized + compressed

For anything ‘modern’ this is probably the best choice as it is accelerated. There is probably not enough room for anything complex in 512 bytes (at least on 64 bits) because Clutter GLSL symbols are long named.

Didn’t try the 32 bits version but this probably save some more bytes.

A good source of fragment shaders to learn from is Shadertoy

Framebuffer + pure assembly

64 bits assembly program that use the framebuffer device /dev/fb0 (fbdev) to output graphics data. (white centered pixel)

This respect the ELF specification; better gains (~36 bytes) can be achieved by overlapping ELF headers or switching to 32 bits. See tiny ELF

64 bits ELF result :

  • 196 bytes
  • 173 bytes compressed

This is only ~9 bytes less than the C version!

More

Some more bytes can be gained by tweaking the ELF header, automatic tools exists to do that.

Lobe.ai – Machine learning made easy

Lobe has everything you need to bring your machine learning ideas to life. Just show it examples of what you want it to learn, and it automatically trains a custom machine learning model that can be shipped in your app.

Easy to Use

Designed to be easy enough for anyone to use. No code or experience required.

Free and Private

Train for free on your own computer without uploading your data to the cloud.

Ship Anywhere

Available for Mac and Windows. Export your model and ship it on any platform.

Lobe will automatically select the right machine learning architecture for your project. Image classification is available now, with more templates coming soon.

The Only 5 Things You Can Invest In

There are only two basic things I care about in terms of how my children turn out someday:

(1) I want them to be healthy and happy

(2) I want them to be good people

In my book, “You’re a good person” is one of the highest compliments you can pay someone.

The world lost a good person this week. Jon Boorman passed away from cancer. He was just a good person and that’s not something you can say about everyone these days.

A number of people shared a piece Jon wrote a few years ago about some things he learned over the years. This one stuck out to me:

When you’re young, you have so much time but never enough money. When you’re old you have money but never enough time.

How you perceive and value time and money will change many times throughout your life, but at the end there’s only one you’ll want more of, would give anything for, but it won’t be available at any price. Cherish it while you can.

In the investment world so much time and energy is spent thinking through various market scenarios, investment strategies, economic datapoints, asset class combinations and financial minutiae. Money rules all else when it comes to what people pay attention to.

I understand why this is the case. Money may not automatically make you happier but it sure can make life easier.

But your capital is just one thing you can invest in. As Jon so eloquently pointed out, time is also an investment and it doesn’t always have to cost anything to invest your time.

With this idea in mind, here are the five main factors in life you can invest in:

1. Your Time. Time is the one asset where there’s no inequality on a daily basis. Everyone deals with the same number of hours. How you manage that time can have an enormous impact on your life satisfaction.

How are you being compensated for your time? Are you able to spend time on the things that are important to you? Are you finding enough time to better yourself or your career?

Who you spend your time with, what you spend your time on, and where you spend your time are some of the most important decisions you can make in life.

This is true if you have a lot of money or not.

2. Your Money. Investing your money is certainly important if you ever hope to gain more independence over your time but this is not solely about the markets.

What do you spend your money on? Who do you spend your money on? How does your spending impact your happiness? What are you investing for? Are you buying stuff or experiences?

Mr. Carson on Downton Abbey once remarked, “The business of life is the acquisition of memories. In the end, that’s all there is.”

Money can help facilitate these acquisitions if used correctly.

3. Your Energy. Almost all of my friends have kids. Many of them talk about how tired and exhausted they are all the time. With smartphones, emails and constant notifications it’s almost impossible to leave your work at the office these days (especially when the office is home for many right now).

People wear I AM SO BUSY as a badge of honor.

Between careers, friends, family, working out, Netflix and hobbies it’s not always easy to get everything in that you need to for a balanced life. Time management is important but so is energy management.

I’ve learned over the years one of the best ways to approach this is the things you don’t invest your energy on. This could be avoiding friendships with people who are constantly negative or toxic. Or maybe avoiding clients or workplaces that will suck your energy dry. Or bad habits that make you feel lethargic, tired or stressed.

The people, work and activities you don’t invest your energy in can help you focus on what really matters.

4. Yourself. If I was a lifehack guy I would say investing in yourself is the best investment you can make but I’m not so I won’t go that far.

But you’ll never have enough energy if you don’t invest in your health. You’ll never advance in your career if you don’t invest in your education and learning.

5. Your Contentment. Happiness is a valid goal but I’ve come to the conclusion that simply being content is a worthwhile pursuit. It could be contentment in yourself, your station in life, the work you do, the people you’re with, the activities you spend your time on, etc.

Being grateful for what you have is a tricky mental game because it’s difficult to strike a balance between wanting to move up in the world while also being happy with what you have.

It’s hard to get out of your own head sometimes.

Some people meditate. Others do yoga. Some people turn to substances. Therapy works for some as well. Working out has always been my release valve.

Whatever it is, figuring out how to keep your sanity, maybe now more than ever, is a worthwhile investment.

Further Reading:
The Bear Market in Happiness

Check out this video made for Jon as well:
Rest in Peace, Rockstar

 

Print Friendly, PDF & Email

Will Avoiding Lattes Make Me Rich?

Many of us tend to spend a few bucks every day on something we really enjoy. It might be a cup of latte from the nearest coffee place, a fancy sandwich from that delicatessen, or a yummy chocolate bar.

But next time you place your order, you might be asking whether your future self will thank you for that purchase. Saving just a few dollars every day on unnecessary spending could make a serious difference to your financial future. It could be the difference between retiring comfortably or not at all.

Giving up your daily coffee spend is often touted by finance experts as an easy way to build a nest egg, but does it really work? Looking at the basic math, spending $3 a day adds up to $90 a month, which is $1,095 a year. It’s a decent sum you could be saving.

At the heart of this strategy is the idea of giving up a pleasure today to benefit yourself years down the line; it’s a concept that many people struggle to grasp. Sacrificing something that’s going to make you happy right now for the chance of a few more pennies in your pension pot doesn’t seem like an appealing trade. We prioritize our current self over our future self. In essence, we prefer to spend the money in the present moment rather than saving it for some vague future goal that we find difficult to imagine.

Is That Coffee Necessary?
There are few things in life that are absolutely necessary. Paying rent and your bills are two of those, buying yourself a coffee is–unfortunately–not. Yet we keep buying coffees.  

Sarah Newcomb, behavioral economist for Morningstar, says any time we decide between two things, we picture the scenario in our mind: We think about what will happen if we do A, and what if we do B instead.

But there’s a booby trap in our mental models, says Newcomb; something called psychological distance. “It’s how we measure the ‘space’ where ideas live,” she explains. “Any time you imagine an outcome, that outcome takes place in psychological space.”

And in that space, things that are far away in the future, such as saving or investing for a long-term goal such as retirement, look smaller and less important to us than things that are closer, such as a hot cup of coffee right now.

So how can we beat our behavioral biases?

Set a goal. It’s hard not to prioritize your present-day self when your future self hasn’t made any plans. Set some savings goals so you have something real to work toward, whether it’s putting money aside for a house deposit or increasing your pension contributions. Having a tangible target makes saving much easier.

Do the math. That $3 spent today may not seem like a lot, but if you add it up over the long term you start to get a feel for the impact it could have. That $3 a day becomes $1,095 a year, and if you invest that and it grows at 5%, after 10 years you’ll have $14,228. Just because you didn’t buy that coffee.

Use nudges. Technology makes a lot of things in life easier, and saving is no different. A number of banking apps now offer to automatically round up your spending to the nearest dollar and put the difference into your savings account. So, if you spend $2.88, for example, the purchase is rounded up to $3 with the extra $0.12. It doesn’t sound like much, but over a year it adds up.

Beating Your Biases
Mike Coop, portfolio manager at Morningstar Investment Management, says, “Not buying a coffee every time you want is deferring gratification, but above all, it is a test for people to self-discipline.”

He suggests that rather than giving up little luxuries entirely, that it’s better to think of them as rewards for a special occasion rather than part of your daily routine. “That way you will enjoy it more and it will make you pause before buying anything unnecessary,” says Coop. Even if you buy one coffee a week, rather than one a day, that’s $78 saved each month.

He adds, “What does not seem a lot today, such as investing just $3 per day, could build up to a decent sum in the future.”

Indeed, compound interest is your greatest ally when saving little and often. Given time, the phenomenon and earning interest on your interest can supercharge your savings pot. So the earlier you start saving, the less you have to set aside.

That $78 invested each month over 40 years, grown at a rate of 6% a year, could grow to a sizable pot of more than $156,000. And that’s something your future self will definitely thank you for.

CloudOffix – OS for Your Business

CloudOffix is an ultimate customer engagement suite. Bridge the gap between the units in your company from sales to marketing, accounting to project management, or helpdesk to HR with a fully integrated CloudOffix platform.

With CloudOffix, you do not need to buy separate business applications for all of your needs.

Try now or Watch Video

Rethinking Coding and Debugging

In the last article I wrote about how I moved from “Too many bugs in C code”, over “This new programming language is fun” to “I will build a platform where users will build & share features”. In this article, I wanna write about a better approach for navigating code and debugging.

Progress

There are two things which have improved software development in the last few decades:

(a) Better programming languages. So we can have fewer bugs, especially “hard to find” bugs. The big question here: is it possible to push it significantly further?

(b) Better IDEs. I believe there is a lot of space for improvements!

There is a new programming language maybe every month, but how often is there a new IDE? Few years? Building IDE is hard. Even that simple idea of that I will show you in this article took me a week to implement.

IDE is a place which brings code, compiler and debugger closer to the user. User experience for writing and understanding the code is super important. It’s a place where people like me spend a big part of their lives.

Modern “IDE”

Lately, We have a lot of No/Low code platforms rising up. They call them Productivity tools, but they are basically IDEs(Integrated Development Environments). In a most extreme cases it’s simple drag & drop blocks or nodes on canvas. The problem is those platforms restrict you with a few specific tools and places where you can apply them. They are optimized for few use-cases. Compromise and you get easy to use. That’s the name of the game.

In the last ~5 years, I did a lot of research and development(many MVPs) in the no-code area and it’s definitely possible to make a general programming environment without writing text/code, but doing that you will be slow, very slow.

Text matters

I believe that the most efficient way how to explain to the computer what to do is through text! The learning curve for writing code is crazy. I’m not surprised that so many people give up. There has to be some invisible force, keeping your motivation up enough to continue.

I’m not a big fan of pure text programming. I like IDEs with debuggers, refactoring, REPL, auto-complete, fast code navigation, etc.. I’m a huge believer in low-code, in terms of that everything is code, but you don’t have to write all of it. Using the keyboard in all situations doesn’t make sense: picking a color is better than writing “120, 64, 80”, drawing a rectangle with a mouse is better than writing numbers as parameters.

Learning how to write on a keyboard with all fingers without looking takes 20-40hours, but then you will be way more efficient for decades. In the beginning, it’s hard to start coding, but when you “made it”, your efficiency will be out of the roof and you’ll eat no-code programmers for breakfast.

I get it, text is not sexy. Try to imagine super happy & smiling people in ads, touching the screen … with code on it … Ouch. But text works!

Ok, no more theory, let’s move to the new stuff.

No jumps

Many say that a code is like a book. You read a book from left to right, top to bottom, another page, another chapter. But with code, it’s not like that. It’s chaos. You start in main() and then you jump from one place to another, forward and back until a call-stack in your brain is full. Coding is more like the 2nd season of Westworld. Non-linear storytelling. You probably don’t like it, unless you really are into solving puzzles.

But what If instead of procedural code, the code would be a stream of lines? You call some function(fill parameters) and the body just shows up below it. So your focus is not disturbed. The advantage is there are no jumps(while reading text), which especially beginners appreciate.

For debugging, there is no need for a call-stack(a list which shows you jump in the code). Normally, setting a break-point is not easy, because the program can visit the line through multiple paths(function can be a call from different places), so you spend time debugging that path to the line. In SkyAlt, you just unfold the particular function call and see the body, so debugging gets much easier.

I’m testing it only for a few days, but reading, creating and editing code feels really good.

Note: The SkyAlt syntax is adjusted for this kind of folding – you can have only one function call on the line. In many other languages, you can have multiple functions calls on one line(as parameters) and then of course is not possible to show all their bodies in linear text space.

SkyAlt build 3(alpha).

Visual REPL

Programmers talk about the accidental complexity of programming languages all the time, but what about the accidental complexity of IDEs. What do you do when you see a bug in GUI? Do you think which class is it? And where will you set the breaking point? Or do you just press alt-key and click into GUI? Because the last option is SkyAlt way. One-click into GUI. Bum. You debugging the line of code, which draws that part.

SkyAlt execution model is ImGui(Immediate mode GUI), so every frame starts by calling Main() function with a clean frame buffer. When you write code, SkyAlt has preview panel which shows you how the app looks on the line which you’re working on. Of course, the preview is live so you can click/write into it, but the interpreter will stop when your current line is hit and starts another frame calling Main() again. When you creating code, it’s easier to catch bugs than using some reactive model, when something triggers something and so on. Check the video.

Note: I originally think that the alt+click idea was mine, but later I realized that a few years back I saw it in Bret Victor’s Inventing on Principle. Although, my implementation is a little more advanced because SkyAlt will show you the whole “call stack”. In his video he probably just save line number to pixels, so you don’t know from which place the function was called.

SkyAlt build 3(alpha). In the video I alt-clicked on “Show all” button and later on “X” button.

Summary

Making only programming languages is not enough. We need more IDEs. Different approaches how to create and navigate code faster.

If you don’t wanna learn how to code, because there is no-code, you will never be as free and efficient as people writing code.

I presented the new way how to create code without jumping through text. This approach nicely fits for live debugging(no call stack) as well. Also If you alt + click to program’s GUI, SkyAlt will show the part of the code where that GUI was made. It’s a huge time saver.

I really love SkyAlt and I wanna build every new idea I have in it. I don’t wanna program in other languages and IDEs anymore.

Milan Suk
26th October, 2020