https://www.rci.com is an important one for me.
1 Like
Ok, @hostilefork is reworking the TLS scheme to make it more readable prior to updating it. But he's living out of his car out the moment so it might be paused while he's finding new digs.
And https://www.rci.com doesn't support TLS 1.0 which is why ren-c can't read it.
Though I don't like taking people's money, it wouldn't hurt to have some. 
But bounties are difficult things...this article covers a lot of the points:
http://www.dit.upm.es/~jantonio/documentos/articulos/bounty.html
Financial incentives aside, I do appreciate that you have supported the development a lot as it is! So I had sifted around some about a week ago to understand the situation with TLS well enough to know what's involved.
That's from the perspective of "what the heck is with these different versions anyway", along with some amount of familiarity with what exactly it means to "support ECHDE" (e.g. there are a ton of different ways to calculate even thingsl like an 8-bit CRC, exactly how much code and variations of it would the binary need?)
So...what's up?
"Black Box" Implementations
One thing that feels like it may be a "bad" way to spend time would be to make an extension that does a simple, synchronous, TLS-aware READ and WRITE. I say it is bad, because that would be putting the TLS functionality into a black box...which makes it basically useless if you were writing some more interesting client or server program, holding a connection across several separate reads and writes.
But such black boxes exist...like libCurl (which really just gives you the services of curl without having to ship a separate EXE or exchange data through intermediate files). But on Windows, you don't even need that: the OS has the (IHttpRequest) built in. It's not rocket science to invoke...though it does use COM, which is a little ugly when called from C. Red's simple-io uses it, see %red/runtime/simple-io.reds
I'm not exactly clear on what blocks people from active experimentation. It's hard for me to understand if such a stopgap measure would really move the needle for anyone. Is calling out to curl, e.g. with @rgchris's curl wrapper, not sufficient to get started (assuming such things are brought up to date)?
Building on Cyphre's TLS
The TLS code was a bit of a nightmare for me, because if a core change ever broke HTTPS I was generally in for a world of pain. I had mostly no idea what was going on with it, and spent more time looking back at what behavior I might have caused to deviate vs. actually trying to understand the code.
Now looking over it and sort of getting the direction, I'd say it's not so bad. It was done largely in the spirit of how one might think a Rebol-based program should try to make a kind of "executable spec". So it has state transitions that look like what's described in the protocol, and makes a decent amount of sense. It's not a bad place to start from.
But as a small example of the aspect of it that was bad, look at this bit of code for "set the correct message lengths":
change at ctx/msg beg + 7 to-bin len: length? at ctx/msg beg + 10 3
change at ctx/msg beg + 4 to-bin len + 4 2
append clear ctx/handshake-messages copy at ctx/msg beg + 6
return ctx/msg
Does that look like "code of the future"? Not to me. :-/ What it's doing is that it has a packet structure, but some of the bytes are related to the size of sections in that packet. So it's easier to answer questions about what goes in those bytes after the whole header is formed. But it's using hardcoded numbers to do it.
The reason I picked this is because that's something I tweaked with a tiny dialect. I made it so that the binary spec could have SET-WORD! in it, and those SET-WORD!s recorded the position in the binary development, so you could go back and reference the values in based on ssl-record-length:, message-length: and ssl-record: in the block:
change ssl-record-length (to-bin (length of ssl-record) 2)
change message-length (to-bin (length of message) 3)
append ctx/handshake-messages ssl-record
Writing the dialect took only a minute. Figuring out that's what the original code was doing is what took the time. :-/
But like I said, I don't think the fundamental approach is bad. It's actually a short file, and I think we can keep making it shorter and better. It can be an opportunity to exercise Rebol and show something different...a TLS protocol running on an executable spec.
What does "TLS 1.2" support mean, anyway?
One thing you find pretty soon is that the spec is kind of big, and people make lists of the features or concerns they don't attend to.
Speaking of things that aren't attended to... people might not be aware that there's no "S" in Rebol's HTTPS. It extracts a certificate and then...takes its word. So saying that it has an "https implementation" is really saying "it will talk to anyone who hands it any certificate and encrypt the communication taking its word".
Doing certificate validation is...kind of...important? Anyway, that's an undeveloped area which would need attention.
Using a C TLS
If one is not going with a "black box", then one could imagine hooking up a Rebol-based HTTP or TCP server scheme on top of a C-level TLS implementation. It would appear native, the way that the TCP scheme does today.
It would be faster...but more importantly, it would be written and already support 1.0/1.1/1.2. Someone else would be responsible for maintaining it.
There are various options out there. People make OpenSSL out to be somewhat beastly, but it's documented and well-known. @gchiu has pointed out stuff like TLSe, which is a "Single C file TLS 1.2 (also 1.1 and 1.0) implementation, using libtomcrypt as crypto library"...very much in line with Rebol's stated mission.
If there were some kind of rush and people needed working TLS on a short-term timescale, I'd say this is the more reliable option. But the only thing to splice it into right-now-today would be the Rebol PORT! model, as a swap-in for a TCP port. And if the only usages people are imagining are HTTPS read and HTTPS posts, vs. writing some kind of interesting client/server with HTTPS needs... it doesn't make sense to do that rush instead of the black box!
Crypto
TLS is just a protocol for two parties to agree on how to talk. When they do, they use a group of secure hashing and encryption functions. A client saying "hello" to a server offers it a list of what it can support, and the server looks through the list and picks the one it prefers.
One issue is when the server says it doesn't even like the client protocol itself. ("You said hello wrong".) So TLS 1.0 isn't being supported anymore. But the other is, that even when it is...or when you update the protocol, you still might not be able to speak a "cipher suite" that the server will accept.
Whether one sticks with the Rebol TLS or uses a C one, you still have to worry about having ciphers. Linking in new ones isn't exactly rocket science. You can get them various places...and people have brought up libSodium, which offers a "curve" which is compatible with ECHDE.
One edge C TLSes have is that they are often already geared up to use popular ciphers. So they are lined up in a "turnkey" way to just set a compiler option and use it, all the interface code for getting bytes back and forth from the cipher to the protocol has already been written by some poor soul. But for Rebol, each cipher implementation has to be wired up to read a BINARY! and and write a BINARY! out, etc.
Anyway it's not impossible, and it has been getting better rapidly due to libRebol. If you look at some of the code in %mod-crypt.c and compare it to the state of things in Saphir when it had to go through the RL_API, it's just night and day clearer--and more fault-tolerant. Still it's work that has to be done.
Conclusion
Previously I kept the %prot-tls.c around mostly because read https://example.com was one mean piece of test code, running a lot of stuff. If you asked me if it had a future, I'd have said probably not...and that it would likely be swapped out for some commodity C code at some point.
But I now think it probably does have a purpose and is relevant in telling the story of the language, and demonstrating its abilities. It does have a long way to go, though...as I mention, the certificate validation is a pretty big deal.
So it would be interesting if Cyphre wanted to come back and give it a revamp. However, there's a lot of friction with some other old-time Rebol people over the fact that I fix things that are broken (and yes, sometimes shuffle things around exploratorily while searching for those fixes, which is necessary). So, there's that. But as a bigger issue--I don't really know that a TLS of any kind, C or Rebol, has much purpose beyond a black box until the ports have more sorted out about them.
So on that note: If people showed me working programs they care about that were in Ren-C, that relied on calling out to curl.exe, and that there was a good reason why eliminating that curl dependency was important, I could consider that a reason to be compelled to do the black box versions. Getting https READ and POST to work on Windows via calling OS services would take a little time to write and debug, but not be terribly hard. libCurl is easy to compile with and link on linuxes for that functionality, also.
I can use (external) Curl for my short-term needs. And medium-term Curl could be embedded in Ren-C, as Red does. I do feel that Ren-C should handle common web protocols seamlessly, otherwise it will cease to be a "batteries included" network-ready language.
Ideally a solution would be part of Beta/One. Hopefully without stealing focus from more important features.
Happy to revise bounty terms/details and to venmo working solutions which are Ren-C worthy.
I don't have any TLS sites at present that I can't reach with r3/hf. But I imagine that it's just a matter of time.
So, my thought would be we need to aim to support being a https server using TLS 1.2. Certificate authentication can be an option in the client. I have seen the way 8th does it and it's acceptable to me.
So Brian has implemented TLS 1.1 and 1.2. Time to test all your favourite sites!
2 Likes
Well this is exciting! I will take a look asap and follow up.
Grrr, company Semantec Endpoint Protection is blocking me from using 0.3.40/r3-bc3dda1-debug.exe
Is that something where you can't override it? :-/
Perhaps you could use a virtual machine.
I'll have to see if I can get permission to run it as Admin.
You need to run it on internal sites??
I'm talking only about running it on my local machine. I've never had a problem with the other .exe's (apart from Red), but this one is being fully-blocked for me. I'll need to get permissions. Currently this is my only working machine, although I'll have a home lenovo mini-desktop back in order soon.
Just circling back to this.
While I haven't been able to work on this and test the functionality, if those of you close to this issue give the thumbs up, I'd be happy to fulfill my $ piece of the award for this bounty.
I can private message Brian to figure out whichever payment method he'd like to settle this.
Given just the rate at which I'm incurring unplanned expenses (e.g. $300 to have my car re-chip-keyed after losing my only remaining key, also losing my glasses :-/) it would be helpful! I am apparently becoming senile in my old age.
However...
...well we can't have that! :-/
Don't feel you have to just because you said you would (especially if you're short on computers to not be able to help development!!!)
For one thing, I don't think the task ever had a completely firm definition of "success" to be measured by (including what a reasonable amount of time to take to implement the request would be). And secondly, you've provided plenty of moral support otherwise, as well as helping keep the forum up.
But as things go---I basically understand the code and the nature of the specification, and can continue to extend it if we have a need. Though it doesn't incorporate an ECHDE curve yet, I'll add it when someone says it's blocking...won't be a problem, it's just there's other things to look at first if no one is blocked by its absence.
Having dug around in a lot of different TLS implementations now, I can say that I do feel there is something Rebol has to demonstrate. While the code can be improved, I'm not so sure that many of the other codebases I've looked at have much of an upside or future--due to the static nature of the languages they're expressed in. I think it was the correct move to update the code as Rebol vs the other alternatives, to have another real-world real-life case to point to and ponder.
Well, we got TLS 1.2, and ECDHE can be done says Brian .. so I think the bounty can be paid.
Thanks,
I did not pledge but I think he well deserved it. I will see to my Paypal account if anything is there so I can donate a symbolic amount.
Ok, if you think hf did enough to deserve the bounty, then send your $$ to metaeducation and like many of us, he's on a gmail dot com account.
@gchiu Graham, your answer here raises some questions for me now. First you state that the bounty can be paid, now you say "if you think" implying that you are not so sure anymore?
So did you pay out the bounty reward to @hostilefork ?
What is the status?
He's just echoing what I say which is that it's voluntary... and those who would like to send me money may do so directly... so he doesn't need to have any treasurer responsibility in the matter.
SO JUST THEN STATE:
If you made a pledge, we think the bounty conditions are fullfilled, so please make sure you fullfill your end of the deal. This is the account to pay to.